Hack and / - Right Command, Wrong Server

It's easy to lose track of what your servers do when they number in the tens or hundreds. Here are a few simple techniques I've found that make it easier to manage them all.

When I first started out in systems administration, I had only a few
machines to keep track of. It was relatively easy to remember which servers
did which functions (mostly because one or two machines did just about
everything). If a server had a problem, I immediately knew everything it
would impact.

For better or worse, nowadays my position has become
more complicated. When you personally manage tens or hundreds of machines,
it can be difficult to keep everything straight. When a server goes down,
you might no longer know what services are impacted or who else to notify.
Beyond that, there's also the dreaded
running-the-right-command-on-the-wrong-server mistake. I think every
sysadmin has typed halt, rm
-rf or some other destructive command in the
wrong terminal at least once (just ask my old boss Bill).

In this column,
I discuss some methods I've found to help you keep track of your
servers. Although I can't guarantee you'll never type a command on the wrong
server, I can say that as your environment grows to hundreds of servers, these
techniques will help you pick up where your brain left off.

Message of the Day

The message of the day (motd) is the message that greets you every time you
log in to your system on the command line. For instance, here is the message
of the day on one of my old Debian servers:

Linux napoleon 2.6.20-1-k7 #1 SMP Tue Apr 24 22:37:29 UTC 2007 i686
The programs included with the Debian GNU/Linux system are free
software; the exact distribution terms for each program are
described in the individual files in /usr/share/doc/*/copyright.
Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent
permitted by applicable law.
No mail.

Messages like this are pretty generic, so it's easy to take them for granted
and leave them alone. After all, in this example, I already know the OS,
hostname and kernel version (Linux, napoleon, 2.6.20-1-k7). You can extend
this information, however, and list anything you want.

The message of the day is managed in a file called /etc/motd. It's a simple
text file, so you can modify it to say anything you want, although you'll
want to limit it to what can fit on a standard console screen. Note that on
modern Debian-based systems, the /etc/motd file is somewhat dynamic, so you
will want to modify /etc/motd.tail instead.

So, how can you use this file to your advantage? A lot of security-minded
administrators add a special terms of use in this file to note that their
systems are private and do not allow unauthorized access. In that case, the
motd acts like a No Trespassing sign, so if someone hacks in to the system,
law enforcement has help demonstrating that the attacker was notified that it
was a private system.

Although you may or may not want to add a No Trespassing sign to your motd,
there are a number of other things you can add to the motd to make your
life as an admin simpler. For instance, you could add a short set of
documentation about the server, including what the server does, other groups
to contact if there is a problem on the machine and even any special
locations where custom files are stored. That way, when you log in, instead
of a boring default motd, you could get something more like:

You even might want to use the motd to pass along useful tips to regular
users on the system. For instance, let's say your users use vim to
view log files. On some systems, vim stores a complete copy of any files
you open in /tmp. Although that's fine for a small text file, when you have
users opening 1GB+ Apache logs, your /tmp space fills up quickly, and you
are paged again and again. One solution might be to add a gentle reminder
in your motd to use less, not vim, to read large text files.

Tweaked Shell Prompts

Another great way to help remind you which servers you are on is to tweak
your shell prompt. If you are a good security-minded admin and become
root only when necessary, a quick tip is to make the root prompt a
different color (like red), so it stands out and reminds you that
everything you do is with root privileges.

There are many different tastes when it comes to a custom shell prompt, so
you might want to tweak this to suit your preferences. Also, I'm assuming you will
be using the bash shell that most systems tend to default to these days, so
the file you should edit is /root/.bashrc. What shows up in your
prompt is defined by the PS1 environment variable, so if you are curious
what it is set to by default, simply type:

root@napoleon:~# echo $PS1
\u@\h:\w\$

In this example, you have a very basic prompt that lists the current user
(\u), the @ symbol, the hostname (\h), a colon, the current working
directory (\w) and a # symbol (if I'm root), or a $ otherwise (\$). On my
sample system, it would look like root@napoleon:~# when I log in as root.

There are plenty of other ways you can tweak the prompt, and if you are
curious, the full list of aliases you can use for it is found in the bash
man page—just search for PS1.

Because I'm focused on colorizing the prompt and not necessarily changing the
format, I mostly will leave the prompt as is. There are a few ways to
colorize the prompt, but the simplest way I've found is to define some of the
potential colors you'd like to use in environment variables ahead of time,
and then you can assign them to the PS1 variable without going cross-eyed
from all the escape characters. Open up /root/.bashrc, and if PS1 already
is defined, add these lines above it:

Now that all the colors are defined, I simply can define PS1 with the
default settings, only with these color settings around it:

PS1 = "$RED\u@\h:\w\$$NORMAL"

Once you save the changes to .bashrc, the next time you log in, you will
notice your prompt is colorized. Now you can spend the rest of the
afternoon tweaking the prompt with different sets of colors and symbols
like I did the first time I found out about it. It even might be worthwhile
to use a different prompt color scheme for different types of servers.

Kyle Rankin is a director of engineering operations in the San Francisco Bay Area, the author of a number of books including DevOps Troubleshooting and The Official Ubuntu Server Book, and is a columnist for Linux Journal.

As Linux continues to play an ever increasing role in corporate data centers and institutions, ensuring the integrity and protection of these systems must be a priority. With 60% of the world's websites and an increasing share of organization's mission-critical workloads running on Linux, failing to stop malware and other advanced threats on Linux can increasingly impact an organization's reputation and bottom line.

Most companies incorporate backup procedures for critical data, which can be restored quickly if a loss occurs. However, fewer companies are prepared for catastrophic system failures, in which they lose all data, the entire operating system, applications, settings, patches and more, reducing their system(s) to “bare metal.” After all, before data can be restored to a system, there must be a system to restore it to.

In this one hour webinar, learn how to enhance your existing backup strategies for better disaster recovery preparedness using Storix System Backup Administrator (SBAdmin), a highly flexible bare-metal recovery solution for UNIX and Linux systems.