Published

Don’t touch the defaults: The Hippocratic Oath for System Administration

The Hippocratic Oath, taken by doctors, is often paraphrased as “first, do no harm”. I’d like to note the broad applicability of the Hippocratic Oath to system administration. What, for example, is the first thing that you do when receiving a Severity 1 incident in the middle of the night? Correct answer: Look around at your surroundings but don’t change a bloody thing. No good has ever come to a system administrator whose first response is to pound the keyboard furiously and poke around systems, making random changes. This isn’t Hackers, after all. Like a doctor, you need a hypothesis as to what is wrong before operating on your patient.

Beyond just incident handling, though, I’d like to apply the Hippocratic Oath for System Administration using a philosophy that I call, somewhat uncreatively, leaving the defaults the hell alone. Too often I’ve seen people randomly change configuration settings without a true understanding of why. This behavior applies not only to system administrators randomly tuning Linux kernel parameters or PostgreSQL postgresql.conf settings without rhyme or reason. It also applies to the way in which applications are set up in the first place. In this post, I’ll give you some of my guiding principles for leaving things alone and just accepting the defaults even if you know they might not be optimal. In other words, leave the unnecessary tuning for your car, not your job.

Distro Packages Are Your Friends

You probably use an operating system distribution. If you need to install Apache, what’s the first thing you type?

If your answer isn’t apt-get install apache2 or yum install httpd, and you don’t have a really, really good reason for not doing that, this principle’s for you.

Seriously folks, it’s 2014. Why do you care so much about compiling Apache from source? The reason software exists as binary RPMs or DEBs or whatever is because it makes life easier and faster. And your package manager solves all dependencies for you. I really hope that this point’s not controversial; people seem to hate on distro packages a lot, but they are mostly quite good and let you stop worrying about args to pass to ./configure and what CFLAGS you need and all that crap.

Also, distro packages have sensible places for where they put config files, working files, data directories, log files, and so on. Why make your life difficult by changing things for no good reason? Why confuse your co-workers? Sure, I’m not a huge fan of RedHat dropping config files into /etc/httpd/conf.d, but I’m not particularly inclined to fight it by moving all of ServerRoot and DocumentRoot to /srv/apache or whatever my pet directory location is.

Understand What the Package Provides You

For this one I’ll give you a worked example. I can’t count the number of times I’ve talked to people who are running Tomcat on RHEL/CentOS/Fedora who go and cook up their own solution for running multiple Tomcat instances on the same machine. If you read the mechanics of how RHEL has set up the init scripts, sysconfig files, etc. it becomes readily apparent that this is a fully supported configuration, without you needing to bake your own with random Tomcat tarballs unpacked all over the filesystem. Here’s how:

Now, plug the framework RedHat already gave you for running multiple instances of Tomcat:

sudo cp /etc/sysconfig/tomcat6 /etc/sysconfig/tomcat6-myco

sudoedit /etc/sysconfig/tomcat6-myco (change your CATALINA_HOME and three other variables to point to /myco/tomcat now)

sudo ln -s /etc/init.d/tomcat6 /etc/init.d/tomcat6-myco

sudo /sbin/service start tomcat6-myco

And you’re up and running! Notice how the tomcat6-myco name by which you started Tomcat implies the sysconfig file to use? Notice how by symlinking the right things from /usr/share/tomcat6 into /myco/tomcat6/{bin,lib} you now get all future upgrades of the main Tomcat package?

Also, much of IT work is just pattern recognition: take what you know already, connect a few dots together, and extend that knowledge to create a hypothesis about a new situation. Suppose I want to do the same for JBoss. I don’t know anything about JBoss but I know Tomcat. I also know that the Tomcat RPMs in RHEL have this functionality, and that the framework works in the way I just described. So should I be shocked that RedHat’s JBoss RPMs work the same way? Hell, no!

Leave Those Defaults Alone Unless You Know Why You’re Touching Them

I have worked with a few system administrators who insist on touching a bunch of sysctls in /etc/sysctl.conf, or MySQL tunables in /etc/my.cnf. Oftentimes they don’t even have any idea why they’re touching them; it’s just a matter of cargo-culting. Is there really a reason, or is it a force of habit?

95% of the software out there has sensible defaults. Most of the distro packages’ settings are pretty good. So leave things well enough alone unless you really understand what you’re changing and the reasons behind it. Sure, the default postgresql.conf doesn’t come with the most optimal settings, but it functionally works. Before optimizing the settings, understand what the impacts are and what each setting does.

Conclusions

Here are some guiding principles for how I run my systems. I run things pretty vanilla, ’cause vanilla is boring but boring means you don’t get woken up in the middle of the night for some random accident that occurred because you touched a parameter for no reason.

Install all OSes using the base package list. On RedHat/Fedora that’s not even the Base package set but something like what we have over in Bento‘s ks.cfg’s.

Use configuration management (Chef, in my case, because I work there) to bring the box up from the base system to the desired state. Everything goes in Chef. No exceptions!

Leave SELinux on but in targeted mode. Sure, SELinux can be a pain in the ass, but targeted seems to strike the right balance between insane lockdown and totally wide open. Besides, it covers off most of the attack surfaces.

Use distribution packages to install software where at all possible. Need more bleeding edge software than your distro provides? Honestly, maybe you’re running the wrong distro: you have an impedance mismatch between the operating system conservativeness you desire and the liberalness of your application stack. You wouldn’t expect to run Node.JS on AIX after all, would you?

Don’t bother tuning anything unless you actually have a problem. Honestly, unless you’re a Fastly trying to squeeze every ounce of performance out of your bare metal Varnish servers, the effort you’ll invest in monkeying with sysctls is just not worth it, especially if you’re running VMs in the cloud where the hypervisor might come along and steal cycles from you anyway.

We used to say that nobody ever got fired for buying IBM, because IBM was the safe, conservative, but functional choice. I argue that even with fancy “web scale” and “cloud” systems, you still want safe, conservative — and functioning. Nobody also ever got fired for treading carefully, thinking before typing, but also leaving things alone when they didn’t need to be changed.

Good article, touches on a lot of ideas. I’d amend it to state, “If you change defaults, you’d better have a good message to your future self as to why it was changed”. Also, you put a LOT of faith in distro maintainers. Might not be as shiny as you think.