Taming the Beast

The right plan can determine the difference between a large-scale system administration nightmare and a good night's sleep for you and your sysadmin team.

Backups

Rethink backups. More than likely, when confronted with a large Linux
deployment, you're going to be dealing with massive amounts of data.
Deciding what data to back up requires careful coordination with
stakeholders. Communicate with users so they understand backup
limitations. Obviously, written policies are a must, but the occasional
e-mail reminder is a good idea as well. As a general rule, you want
to back up only absolutely essential data, such as home directories, unless
requirements dictate otherwise.

Serial Console Access

Although it may seem antiquated, do not underestimate the value of serial
console access to your Linux systems. When you find yourself in a
situation where you can't access a system via SSH or other remote-access
protocol, a good-old serial console potentially could be a lifesaver,
particularly if you manage systems in a remote data center. Equally
important is the ability to power-cycle a machine remotely. Absolutely
nothing is more frustrating than having to drive to the data center at 3am
to push the power button on an unresponsive system.

Many hardware devices exist for power-cycling systems remotely. I've had
good luck with Avocent and APC products, but your mileage may vary. Going
back to our “keep it simple” mantra, no matter what solution you select,
try to standardize one particular brand if possible. More than likely,
you're going to write a wrapper script around your power-cycling utilities,
so you can do things like powercycle node.example.com, and having just
a single hardware type keeps implementation more straightforward.

System Administrators

No matter how good your tools are, a solid system administration team is
essential to managing any large Linux environment effectively. The number
of systems managed by my group has grown from about a dozen Linux nodes eight years ago
to roughly 4,000 today. We currently operate with an approximate
ratio of 500 Linux servers to every one system administrator, and we
do this while maintaining a high level of user satisfaction. This simply
wouldn't be possible without a skilled group of individuals.

When hiring new team members, I look for Linux professionals, not
enthusiasts. What do I mean by that? Many people might view Linux as a
hobby or as a source of entertainment, and that's great! But the people on
my team see things a little differently. To them, Linux is an awesomely
powerful tool—a giant hammer that can be used to solve massive problems. The
professionals on my team are curious and always thinking about more
efficient ways of doing things. In my opinion, the best large-scale
sysadmin is someone who wants to automate any task that needs to be
repeated more than once, and someone who constantly thinks about the big
picture, not just the single piece of the puzzle that they happen to be
working on. Of course, an intimate knowledge of Linux is mandatory, as is a
wide range of other computing skills.

In any large Linux shop, there is going to be a certain amount of mundane,
low-level work that needs to be performed on a daily basis: rebooting hung
systems, replacing failed hard drives and creating new user accounts.
The majority of the time, these routine tasks are better suited to your
junior admins, but it's beneficial for more senior people to be involved
from time to time as they serve as a fresh set of eyes, potentially
identifying areas that can streamlined or automated entirely. Senior admins
should focus on improving system management efficiency, solving difficult
issues and mentoring other team members.

Conclusion

We've touched a few of the areas that make large-scale Linux system
administration challenging. Node installing, configuration management and
monitoring are all particularly important, but you still need reliable
hardware and great people. Managing a large environment can be nerve-racking at
times, but never lose sight of the fact that ultimately, it's just
a bunch of Linux boxes.

Jason Allen is CD/SCF/FEF Department Head at Fermi National Accelerator
Laboratory, which is managed by Fermi Research Alliance, LLC, under
Management and Operating Contract (DE-AC02-07CH11359) with the Department
of Energy. He has been working with Linux professionally for the past 12
years and maintains a system administration blog at savvysysadmin.com.

It's is a virtual drop-in replacement for the venerable Nagios project which seems to have stalled into legacy land. It also has a performant, distributed architecture that includes database support for Postgres and Oracle as well as MySQL. It's pretty sweet and easy to deploy too. Something to watch and try out.

As Linux continues to play an ever increasing role in corporate data centers and institutions, ensuring the integrity and protection of these systems must be a priority. With 60% of the world's websites and an increasing share of organization's mission-critical workloads running on Linux, failing to stop malware and other advanced threats on Linux can increasingly impact an organization's reputation and bottom line.

Most companies incorporate backup procedures for critical data, which can be restored quickly if a loss occurs. However, fewer companies are prepared for catastrophic system failures, in which they lose all data, the entire operating system, applications, settings, patches and more, reducing their system(s) to “bare metal.” After all, before data can be restored to a system, there must be a system to restore it to.

In this one hour webinar, learn how to enhance your existing backup strategies for better disaster recovery preparedness using Storix System Backup Administrator (SBAdmin), a highly flexible bare-metal recovery solution for UNIX and Linux systems.