Over the last year I have run a server using the Linux flavor Gentoo. While most of the servers I deal with these days run FreeBSD (WireLoad’s servers included), I was curious about what speed I could get out of old hardware with Gentoo. Gentoo is a system built to make it easy to compile every little piece of software in the system with the biggest and baddest gcc flags imaginable for your particular hardware. In theory this should lead to a faster system.

The experience has been a bit of a mixed bag. There are things I really like about Gentoo: the package management, USE flags and the sophisticated dependencies system. But unfortunately the drawbacks are severe for a server setting.

The Good

The system is better than most Linux systems I have seen when it comes to general package management and installation. The emerge command is excellent. It makes it easy to update and install software together with the necessary dependencies. A well thought out system called USE flags complement this system well. Rather than having to tinker around with every program you compile, you can set some global USE flags. Most packages will take note and your system will be made into one harmonious whole of software agreeing with each other about what should be used and what should not.

The easiest way to describe the benefits of this is by comparison to a normal FreeBSD server and the installation process. Assume that you don’t have X11 and that you don’t want it. Now, you are installing PHP and you want to have support for graphical operations (image conversion, CAPTCHA generation, etc). So you find the make flags to do this and you add them to your make.conf. You hit ‘make install’. During the build process the ports system goes out to build the dependencies of your graphics library. The graphics library requires fonts. The fonts want a font manager. The font manager is by default configured for X11. Wait, X11? Suddenly, X11 becomes a dependency and you find yourself hitting Ctrl-C rapidly!

In Gentoo on the other hand you would have -x11 in your USE flags and everything would be cool. It’s a slick system.

Colorful Gentoo.

Gentoo also has an active community. As we have mentioned in this blog before, there is an excellent forum for Gentoo users. There are also little touches like how Gentoo uses colors by default, to improve clarity and to go easier on the eyes.

Disadvantages Expressed in Time

All of these things make it a pleasure to use Gentoo. I would love to use it for a desktop some day, should my Apple OS X machine fail me. But I really don’t see myself using Gentoo in a server setup again. Here’s why:

1. Gentoo is Time Consuming to Install
At least when I installed Gentoo there wasn’t really any installer. The documentation is excellent and describes exactly what you need to do, but it takes a while. In fact, it took me several hours to set up my first Gentoo system. And that was just the beginning.

2. Gentoo is Even More Time Consuming to Install
The strength of Gentoo is the compile everything mentality – at least that seems to be the main selling point. Unfortunately on my low-end test server it took about three days to compile even the base system with Apache, MySQL, Python and some other important software. My machine was working non stop compiling things during this time.

I understand that the latest recommendation is to not perform a so called ‘stage-1′ installation anymore. I would recommend following this suggestion. But then what happens to the compile everything advantage?

3. Gentoo’s Stability Strategy: Update Everything
Since it takes a long time to compile a program, you usually don’t want to have to do it too often. Unfortunately Gentoo encourages you to update software on a frequent basis, just for the sake of updating.

There is no ‘stable’ version of Gentoo. Gentoo is rather a moving target where emerge will forever cause your system to approach the cutting edge. From the Gentoo handbook:

From the beginning, Gentoo was designed around the concept of fast, incremental updates.

If all you’re concerned with is keeping your web server up, what you usually want to do is to set up a stable system and then forget about it. You install security updates as needed but that’s it. With Gentoo, this isn’t really feasible because there is no ‘stable’ Gentoo release.

What’s worse, there will on occasion be a sort of ‘system update’. This is called a new ‘profile’. The Gentoo documentation and the handbook will at this time encourage you to update to this new profile. A profile update will try to replace your basic system. If you are a system administrator, rather than a desktop user, this should be enough to scare the living daylights out of you!

A profile update will touch a very large number of configuration files, and it may even alter your startup process. Obviously this is not something you want to do to any server. It would be very difficult to verify that everything works as it used to afterwards, and you’d be fairly likely to end up with broken configuration files that may stop working the next time you reboot. This is in fact exactly what happened to me, despite a substantial time spent updating /etc files. The end result: the machine had to be resuscitated on-site with associated downtime.

For a more sensitive server than my test system, you’ll want to simply retire the system whenever a new profile comes out. Just start over fresh with a new Gentoo installation on an alternate machine and go through the setup process. This way you can be fairly sure it’ll work even after a reboot. Once you’ve verified that everything works, switch to the new system.

4. Gentoo’s Security Strategy: Update Everything
As you might be aware, FreeBSD has a nice little program called portaudit. This utility will alert you if you have any software installed with known security holes. Then you can go ahead and update that software with a simple ‘portupgrade‘ command. There’s rarely any problem with this process.

Now, Gentoo also has something like portupgrade. What it doesn’t have is portaudit.

In all fairness, Gentoo has an experimental command called ‘glsa-check’. This command automatically examines whether your system is affected by vulnerabilities described in Gentoo issued security advisories. It also knows what steps need to be taken to fix a given security issue. I really like this development, but I understand that this command is not considered production ready. The Gentoo manual page about it is filled with warnings that this is a tool under development.

In the meantime, Gentoo rather encourages you to update the whole system. And of course a system wide update tends to cause just the amount of havoc you would expect from it.

Gentoo too Time Consuming and too Risky for Servers

I firmly believe in updating server software only when you need to. If you don’t need new features, and things are working, why change anything? If you update anything you will doubtlessly need to update configuration files. You will need to fix things that break in the upgrade process. This is exactly what happened to me with Gentoo during its test year. I had nearly no idea of what I was updating as I ran the dreaded but most needed “emerge world” update. And once I was done I still no idea. I spent a long time working my way through updates in the /etc folder, using the built in ‘etc-update’ command. I tried to read the enormous emerge log file and take appropriate actions. And still things broke.

The best way to keep a system stable is to get it working and then not changing anything. This is hard with Gentoo. Gentoo wants you to change a lot of stuff. It wants to be bleeding edge.

And hence my conclusion. Gentoo is fun to play with, but oh is it time consuming. I guess that’s the cost of living with a hardcore compile everything attitude – you’ll be on the bleeding edge and you better make sure you can balance on such a thin edge. For a desktop system, Gentoo seems fabulous. Fun to work with, colorful, a beautiful ports like system for software. USE flags.

But for a server, especially a production server, Gentoo just isn’t time effective. It’s both the time it takes to put in security updates, and the time possible reinstalls will take. I believe there were three profile updates for Gentoo in 2006, and very little support for older profiles. If you’re like me you’d probably much rather not reinstall your servers three times a year!

In closing I want to quote something Gentoo told me recently:

* An update to portage is available. It is _highly_ recommended
* that you update portage now, before any other packages are updated.
* Please run ‘emerge portage’ and then update ALL of your
* configuration files.

Call me stressed out but I really can’t fit too many ‘update ALL of your configuration files’ into my schedule. :)

Trackbacks/Pingbacks

135 Comments

I’m not in the server admin business. But I do wonder though. Security patches usually result in an updated release. While some distros backport security patches, doesn’t securing a system mean confusing hackers, and hence changing it often enough that not only do you fix known security flaws, but also unknown vectors? MS has had a bad record at doing this properly, usually opening new holes with every update, but OSS devs have a pretty good track record as far as I know. Just my $0.02

I have been a Gentoo user for about two years and hit all the problems that the author describes and yet I have recently decided to move my home server to it. Why? because I hit all the problems in the first 6 months of learning. Since then my desktop systems have been kept up to date doing sensible upgrades even without the glsa-check commnand (thanks for that, could be useful)

Well, Kudos to the author for exposing (and getting slashdotted!) on a subject which we had known for quite some time, yet never bothered to comment on publicly.

It sounds a lot like not many of these people have ever held a respectable job and were told “absolutely minimal downtime”. Had any of these people been told that this was one of their jobs’ requirements, I’m sure they would find another way to preserve uptime as much as possible – which would be not to update every ten minutes.

Another item that bothers me about the Gentoo crowd is that most believe that they know GCC in and out because they edited a few build flags. Well, I know of a few muscles in a human leg, but I certainly wouldn’t be operating on someone, would I?

In the end, its about choosing the right tool, for the right job. And all those commenters that have said “Oh I’ve been running Gentoo for six years and have never had a problem upgrading!” are either simply lying, or don’t live in the real world – because its not perfect. To not believe this fact, or to ignore it, would be a fallacy.

So, “Over the last year I have run a server using the Linux flavor Gentoo”. A server as in ONE SINGLE machine? and this entitles you o say that a distribution is not good enough for production servers?. I manage about 20 physical machines running xen on Gentoo, and on top of this about 80 virtual machines (gentoo based also). I am administering unix servers for the past 10 years and I find Gentoo to be the most “administrable” system to date. It all depends on your skills, and belive me aplying a general usage manual to a server environment is not a so smart thing to do. There is nothing forcing you to update all packages, you could just upgrade the ones with security problems. The profile upgrade you mention appears every 6 months, a timeframe that follows most distributions release cycles, so think of it as an “upgrade” to a new release. So far you acted like a Micro$f7 case study which decided linux is crap after running a RH on a test machine for a couple of days.

I run Gentoo in quite a number of production servers. Let me illustrate how we addressed the concerns you had (n.b.: this fixes them for us).

First off, installation isn’t all that complicated once you do what every shop does beyond a certain amount of servers: server images. This of course only works if you have a somewhat homogeneous landscape of hardware. But at 1000+ servers, I wouldn’t want anything else, if only because of repair handling.

Second, the problem of stability. With such a large number of machines, you typically get groups of 2-30 machines that do the very same job. Thus, they are identical safe for IP addresses etc. Standard procedure here is to have a staging/test machine that’s not part of the productive set but otherwise identical. For Gentoo, this means all compiling happens there, rounded of with testing as far as the system warrants. Then, binary packages are generated (quickpkg is the tool of choice) and installed on the productive machines. If possible, one machine is updated at a time, so there’s no service disruption. This also makes config handling easy. If the software needs to be upgraded on all machines at once, you do it in a maintenance window.

And here’s the kicker: we don’t do this any different on Debian or other binary distros. Actually, Gentoo is more flexible for us as soon as we compile software ourselves. The portage system and its overlays make it much easier to integrate our own packages (or versions of official packages) with the rest of the system. Running your own Debian package repository can be done (and we do it), but it’s more work, IME.

As for the ricer problem (people using insane gcc flags): nobody is forced to do it. Actually, all official docs discourage aggressive tuning in that area.

Oh, and for those people who run a single dedicated server privately… well, there everything is always a matter of taste. So if you’d like to run LFS there or an age-old Yggdrasil… whatever floats your boat. But saying that Gentoo is unfit for a productive environment isn’t true.

i didn’t even have to read your article, as soon as i saw you run freebsd normally and now tried gentoo on a server i knew exactly where you were comming from

i run at a small web hosting company in sweden and we run mainly freebsd on all our servers, i often see customers come in with dedicated or co-location servers running linsux and i see nothing but problems

i’m sure linsux is a great os for hobby people and general geeks but i have a life and don’t need to update my kernel every week just to get the latest features, freebsd works for me and we still have 4.1, 4.4 and 4.5 machines going strong

patches do come out for freebsd but in most cases you don’t have to apply them, it all depends on what sort of services the machine is running and other circumstances of course

Package maintenance is really one of the big weakness’ in Gentoo. On numerous occasions I’ve got broken packages due to the lack of responsibility of the package maintainer, who ships out packages which aren’t tested thoughrouly.

First FreeBSD is UNIX. Yes FreeBSD runs beter then all other Linux distro’s but FreeBSD lacks any graphical features(stolen linux features doesn’t count) so it is only ment for guys with an old grey beards.

Gentoo is the perfect development system, gentoo is the only system that doesn’t miss any support for any program language! I mean support on a way you do not to be original with an installation.
It is never meant to be an 10 year old server. That’s why for geeks/nerds/professionals (like me) bsd exists and for windows noops redhat has been invented for server use. Gentoo is in my oppinion the only linux system that really can addept to every environment and yes that gives an administrator to much work. Gentoo is ment for professionals that need a system for a really exact environment and think LFS is to much work.

Please take it easy. I’m using gentoo on 6 servers and it’s stable as hell. Of course you MASK the packages like mysql/apache for a webserver and decide yourself when to update.

As for the system/portage updates, all config files are kept aside until you want them updated.

Gentoo is a safe bet if you:
– lock down the major packages for a system
– update daily!!!

Then it’s easier to maintain than any other distribution.

Also look into the command quickpackage that creates a binary package of something installed. So if you upgrade apache on a webserver and it does not work out it takes a second to downgrade again – with your config files. Show me that feature somewhere else.

I would never switch a server back to a system where I have to fully upgrade every few month. Already got my grey hair ;)

I think that author is frustrated with lack of “free will”. He haven’t read whole handbook and other documentation including man-pages he mentioned. I’m running GENTOO on many servers under very strange conditions (DoS…) and It performs MUCH FASTER than ANY OTHER Linux distro or FreeBSD.
Author forgot what does mean FREEDOM and why he does not use proprietary OS.
Sorry for my poor English, I only practice reading…

You Gentoo fan freakazoids need to get one thing straight: SERVER ADMINS DON’T HAVE TIME TO BE CHERRYPICKING FOR UPDATES. Admins need a command that updates everything up to the latest bug-free version of the package that WON’T BREAK THEIR SYSTEM. That usually rules out major or minor version upgrades of packages.

Hence, the usual blabber about “any seasoned admin should check the logs and only upgrade the packages that you want” is just a crock of shit for deadbeat folks who don’t have other things on their schedules. We professionals don’t do that: we only do an “apt-get upgrade” or “smart upgrade” and it works. It WORKS.

On All distros such as Red Hat, Fedora, Slackware, SUSE, Ubuntu and Debian, when you perform a full update (the equivalent of “emerge world”), the system STILL WORKS AFTER THE UPDATE. If I run “smart upgrade” on my system, I am CERTAIN that the system will work 100% fine after the upgrade. You can’t say that about “emerge world” AT ALL.

And no, the fact that you don’t need to reboot your system between major distro revisions is not exclusive to Gentoo. I upgraded FC5 to FC6 with a “smart upgrade” after changing my repos to point to FC6 distro sites… then took my computer to runlevel 1, and back to 5 again, and everything worked just fine. Did reboot, though, because the kernel was upgraded, and there was no way around that.

Oh, you can even change distros with the method outlined above.

And what’s with the compiling? Excuse me, but wasting days of CPU time and disk spins compiling shit just to eke out 2% more of performance is, literally, saving pennies and squandering dollars. Unless you’re building a set top box with a small CompactFlash card as its boot volume, get the fuck out of here, cos the rest of us would rather install a new package in seconds, not hours.

There are a lots of Gentoo users out there, I was one before, it is a good challenge to install “this kind” of linux on your machine. If you have more machines and fast internet acces, probably you won’t have problems waiting for compilation.

The amount of aggressive commenters is astounding. I have to agree with the one who said that the “got a question? RTFM” attitude gives the linux community a bad name. It really does.

Both sides seem to have good points though — if you know what you’re doing (but at what price? reading docs for months even if you have previous experience with other flavours? compiling things for hours-days?) whatever works is fine!

Calling someone an idiot just because he shares his experience with something new he tried is LOW. Somehow I see that much less often from the other linux distros’ communities.

As far as linux goes, I think aptitude (in particular, apt-get doesn’t know that much) can do pretty much the same — holding back essential packages (hello, weird apache configs!), safe-upgrading that doesn’t install or delete anything, and even getting the source and compiling it yourself (although agreeably it’s not optimised for that as well as emerge is). You’re certainly not forced to upgrade to the next version, you /do/ get security updates regularly, and you get stable and well-tested packages. Sounds good for servers to me!

To the BSD commenter above: no, we don’t recompile our kernels every week, thank you very much. Maybe you do that sometimes on a desktop, but you get it right once in a production environment and you’ll only touch it if there’s a serious bug that’s been found within the kernel — surely BSD needs to be updated for reasons like that, time to time; except you personally never need touch the kernel.

And ViPeR, I’ve no bloody idea what you’re babbling about with the graphical support and the programming language support — other distros and the BSDs have that too.

However I still find gentoo’s philosophy appealing. I keep a mandrake system as a home server since 2002, which has been rendered obsolete for some years now and cannot be upgraded to a newer version unless you re-install it from the beginning.

This for example as far as I know could not have happened with a gentoo system.

These topics get way too much publicity IMHO. And there’s a bit too much bashing of “that other distro” (whichever distro that happens to be) in the open source community.

The whole point is that what works for me doesn’t necessarily work for you. You’ve been running the same machines on the same software for the last 5 years? Apache 1.3 doing great on 2.4 and you really don’t need php5, mysql5? More power to ya. Still, it doesn’t make you right for commenting against a distro (that you might not have ever tried) because it actually updates often.

Yes, there are package updates and yes, they break every once in a while. Not unlike most other distros, or at least the ones that have the latest software builds. But the fact that there are updates doesn’t mean you need to install them (immediately or at all). Upgrading only whenever you need it is always a possibility.

Once I was working on a project at home (gentoo), but when I tried to move the project on the live server some of the PHP didn’t work, because of an outdated version. So I go to the sysadmin and let him know, and he says something along the lines of “but the PHP that I have on there was updated recently”. So when I asked “how recently” the answer was “not more than 6 to 8 months ago if I remember correctly”.
Well excuse me for bothering you. My bad, it’s inhuman to expect things to be updated more than once per year or so.
Call me crazy, but I can see no reason (not bordering intense paranoia) against using up-to-date stable software.

I strongly recommend that the author take out the “emerge world” that he seems to have put into his daily crontab. That’s the only thing that might explain his miseries.

There is no other way in hell for Gentoo to update a live and stable system by itself. Gentoo wasn’t to blame for his problems, the author was. He asked for an update, Gentoo obeyed, and then he complains?

“The best way to keep a system stable is to get it working and then not changing anything.” is undoubtedly true. So why did he change it?

Gentoo has lots of areas where it can be improved, but the article completely missed them, and overall just didn’t make any sense.

SERVER ADMINS DON’T HAVE TIME TO BE CHERRYPICKING FOR UPDATES.
[…]
We professionals don’t do that: we only do an “apt-get upgrade” or “smart upgrade” and it works. It WORKS.

I don’t think that a person who can only type “smart upgrade” can be called a professional. My 7 years old son can type that, so he can be called a professional system administrator?
If you rely on “someone other” to provide you the working solution, then it is not you who administer the server. It is “someone other” – RedHat, Microsoft, pick your favourite.

Many, many years ago I learned (when working with Solaris servers) that a production server should NOT have a compiler on it. A production server should have only minimum of what is required for the server to do its work.

Gentoo requires a compiler, Python and many tools not necessary and even dangerous for servers. But then, even Gentoo itself, calls itself a Meta-distribution. For a desktop you may sit on plain Gentoo. For a server you should use Gentoo to prepare binary packages which you use to build a production server (or 10 servers, or as many as you need).

You prepare packages, you test (if you don’t test you are wannabe-administrator, not a professional), and then you deploy.

And for that, Gentoo is good. You decide what packages are compiled and how they are compiled. Gentoo gives you something called overlay, where you may modify the rules some package is built with, if you need that. You have tools to mask packages you don’t want. You have tools detecting a conflict when some package tries to overwrite a file belonging to some other package. You are well-armed.

One of the greatest things about Gentoo is that while Gentoo Developpers prepare the packages, rules and tools for you, you still have the possibility to change almost everything – from USE flags, thru compiler flags and masking packages, to the ebuild scripts – if you only need.

And you, script-kiddies, don’t call the author ‘an idiot’.
This article is well written. There is no lie. This article does not attack Gentoo, there is just a description of one-year experience with Gentoo.
I am using Gentoo since RedHat released its 7.3 version (oh, the pains of reinstall). I really love Gentoo, and I don’t feel offended by this article. Why you do?

I tried Gentoo for about a year. I was just tired of Mandriva (Mandrake at the time) and RPM hell. I am just a hobbyest, so I am not concerned overly with stability. I thought “Hey, this Gentoo/portage will give me ultimate control of everything! I’ll be able to run JUST a server and not have to install X in order to get ImageMagick installed.”

It took 2 days to install on my old P3 server system. Not what I expected, but we’re compiling from source here. Once installed, I thought this was a work of beauty, and I plugged along quite nicely for awhile. Then I REALLY wanted apache2 and php5 installed. Yes, I RTMF’d and got it working within Gentoo’s portage, but in the end it would have been MUCH easier to compile from source on my own (this was before they branched the php tree).

That little event pretty much ended the honeymoon. I was still using Gentoo, but I kept finding little flaws that made using it alot of work. Then the killer was a particular Kernel upgrade that changed something significant enough that I had to keep rebooting the system every couple of days. Yeah, yeah… revert the kernel. I did that on 4 separate upgrades, and that was enough for me to say… “Hey, this isn’t really that stable, I need something that is alot less work.” So, now I run an Ubuntu server distro, and its so very simple and never barfs on me (even with a few “unstables” installed).

I did learn TONS from using Gentoo though. Before Gentoo, I had never even considered delving into the init system, sysctl, compiling my own kernel, etc etc etc. The list goes on and on. This has benefited me so much that if I had another system on which to play around with, I would put Gentoo on it in a heartbeat.

I’ve been running Gentoo for years on my server. After security problems with other distros I picked it because it’s so configurable. I only installed just the
services I needed and absolutely NOTHING
else. I don’t even run inetd. Once I had
it configured and working I DID NOT update
it further. The OS is stable and working.
The security issues are application problems,
not OS problems. I chose apps that were
as secure as I could find, and I update
them when there’s a need not when the
author fixes a bug that doesn’t even
affect me. It’s been stable and free
from breakins since. I think you just
need to rethink how you’re using Gentoo.

I’ve been running gentoo servers for both personal and corporate use for the last ~4 years. Until now I did not encounter any major problem with the system. There is a lot of ressource on gentoo forum and on the internet. There is also a way to protect your configuration easily (in make.conf). I can agree that my first ~5 servers that I’ve setup where a little bit tricky and take me some time to install. Now It take me at most ~4 hours on a standard pc to setup the basic system + first major update and all my personal optimization and basic softwares that I need.

For the time consuming part, yes gentoo take some time, and it’s why you need to schedule maintenance period. This is way more easier to do and more efficient, if you have a problem, you have more time to fix it and if the package don’t compile properly, it wont replace the old version

If I chose gentoo over other distribution is mostly for the availability of the new softwares. Debian has a “stable” version that is ~1 to 2 years old in version (that is quite troublesome) and when you install a new version its really painful with dpkg/aptitude/synaptique/etc… package management to go back to a previous version.

If you’re using Gentoo for the package manager and don’t use cflags or USE tags, you might be better off with something like Debian. The apt system has all the advantages of portage without all the long compile times. Updating every package on the system (apt-get update; apt-get upgrade) takes minutes instead of hours.

I’ve lost a few dhcp servers, a subversion service and a few mysql client applications over the headaches you described.

But I have since overcome them by recognizing a few things – the “move forward or die” mantra of Gentoo is not the Gentoo disto’s contibution to the mix – it’s the lifeline of the security conscious. At some point in the future each of your server applications will be exploitable by a vulnerability.

In order to manage the configuration musical chairs that goes along with this, I’ve created a subversion server and now have a place for each server’s configuration files to lie. If a config file changes the diff is permanently maintained in the repository. And subversion’s client diff tools are far better than terminal based diffs available on gentoo.

So in short, you’ve recognized a headache, but I believe it’s an inevitable one – software updates. The least fun side effect – configuration file merges – may be dealt with with a robust configuration management tool instead of the gentoo provided solution.

I am currently running 4 Gentoo servers and 1 Gentoo workstation on my laptop. I have in the past, ran Slackware and Redhat from around 1996 – 2002 on various personal home computers, Debian from 2002 – 2004 on various workstations, and Ubuntu for three servers in 2005.

I switched from RedHat to Debian (even upgraded one server from Redhat to Debian) because I found myself having to manually determine package dependencies, compiling from SRPM, even tweaking SRPM files to get them to compile and install. At that time, I never heard of yum, and I thought apt-get was the greatest thing in the world.

From around 2004 onward, I started having need for more current versions of server software, such as MySQL 5, or mod_mysql_auth for Apache 2. At that time, Debian development had slowed to a crawl, and the standard was Debian Woody for a long, long time. Towards the end of the period where I used Debian Woody, I found myself having to use Debian package sources from unofficial package sources. And even then, it wasn’t the latest and greatest.

I switched over to Ubuntu thinking that the packages would be updated sooner. I had also started a business at that time, and started putting up production servers using Ubuntu. However, I started running into the same problems with Ubuntu: outdated packages. There was one point when we were developing software using MySQL 5.0’s stored procedures, but there was a bug in the indexing code so our stored procedure would put MySQL 5.0 into an infinite loop, eventually crashing it. I ended up having to compile MySQL 5.0 from source. Also, for some reason, Ubuntu Breezy’s source package compile was screwed up. While I could compile upstream software straight from source, trying to get apt-get to build a package and determine dependencies was a nightmare. It was worse than building RedHat SRPMs.

You see where this is going? My use patterns require a lot of source compile anyways for certain packages. At around March of 2006, we were looking at building an ISP using Xen and the new Opteron chips. We knew that we had to get the most out of each chip. This is where I started looking into Gentoo. I installed Gentoo on a 500mhz machine, and the build time would take three days. However, we built an stage-1 install script. I also modified the script and installed it on a Pentium4 workstation, just to see how it would work out on a workstation.

While it took a long time, because we wrote the install script, most of my later installs were more or less automated. Later, one of our internal production servers running Ubuntu was screwed up by one of the developers, I backed up the data and installed Gentoo on it. The server were were using was a Xen slice on a dual-core Opteron, and the initial compile/install time was extremely fast (a couple hours). MySQL compiled and installed within 15 minutes. The resulting system using the CFLAGs for that dual-core ran at least twice as fast as another slice we had Ubuntu running. Since then, I’ve deployed three more Gentoo slices.

This is not to say we did not have issues. I ran into that network update on the server side. Fortunately for me, since I ran it on my workstation, I knew what that change in network script was for (netplug). On servers, when I run etc-update, I know which configuration I have changed and which ones I havn’t. I generally do the automerge for the ones I have not changed, and otherwise take a look at the ones I have changed — usually a few files. Also, many of the packages are moving towards providing a sensible default and a .example. We did have one issue with the network script, but we were able to get that one resolved quickly.

Another issue is the compile usage on the production servers. Since we’re on slices, the compiles eat up every available CPU resource we have on the slice. It slows down the web serving to a crawl. Since all the production servers run on the same Opteron chips, I’ll be running emerge with the binary package building enabled, build it on one system, and install it on all the other systems.

On the other hand, there’s some things about our setup that makes it a pain to run Ubuntu/Debian. We’ve deployed Ruby on Rails. Ubuntu and Debian, though, insists that the ruby gems and the various plugins and whatnot go through apt-get. There was one period of time where Ubuntu required a downgrade from Ruby 1.8.4 to 1.8.3, which broke the Rails code that required 1.8.4. On our last remaining Ubuntu server, I ended up (guess what?) having to do a source compile on Ruby 1.8.4 just to get a production server running.

When it comes down to it, I run Gentoo because if I’m going to have to compile from source, I want a package manager that can handle that well. And that’s Gentoo.

Nice essay and you hit on many points that I consider to be important issues with using Gentoo as a server. A couple of years ago, I moved to using Debian for my home hosting scenarios and Gentoo for my desktop. I’ve found that to be the ideal combination. Debian provides me with a slow upgrade cycle, super-stable milestone builds, and a fabulous package management system. I will definitely say that emerge is on track to being as good as apt, and this is quite a compliment. I’ve long felt that apt was the only package management system that offers something better than M$ (/ducks). The onsite rebuilt of a Gentoo server and it’s associated downtime is exaclty keeps me from trying to use Gentoo as a server.

Wow, what a troll. Next time, try using the distro for more than a week before you slag on it. You’re saying I shouldn’t use Gentoo on a server because it takes too long to install, and when you update software, you have to update everything? Argh, fine, rather than be a troll myself, I’ll answer his two points:

Yes, it takes longer to install than other distro’s, that’s because you’re compiling every package from source. Duh. If you want quick, use an rpm distro. If you want optimization, use Gentoo. Gentoo has never claimed to be a fast install.

No, you don’t have to update everything at once. You can choose to update only certain packages, or use other utilities just to install security updates. RTFM! Hell, nobody said you had to update anything at all!

And I’ll throw in a few things about Gentoo that makes it shine on servers:
* Customization: You can have complete control over what software gets installed, and what doesn’t. Drives me crazy every time I see a headless RH server with kdelibs or qt installed on it.
* Optimization: Why in the hell are using software compiled for a 386 on a Pentium 4 class machine?
* Documentation: Gentoo has some of the best, if not the best, documentation of any distro. And the forums are an unbelievable resource as well. I use them even when troubleshooting other distros.

Yes, I’m a Gentoo user and it is my favorite distro. No, it’s not for everyone, but don’t slag on it because it doesn’t work just like your favorite.

I think the author of the article and 3/4 of the people who replied to this article failed to see the forest for the trees. Gentoo is an excellent PLATFORM to stage your environments. If you don’t understand something it does not mean it is not an excellent system. Gentoo is not for your average “Weee I ran apt-get now im updated and ‘secure’ yaaaaaaay” type admins. It is an advanced distribution which offers maximum flexibility for nearly infinite scenarios. You can lock versions of installed software, you can build binary packages from ebuilds for deployment in production environments, you can tweak build options and dependencies without much hassle, you can basically do anything you need to make a linux box do. As far as security goes, Gentoo is a dream. My point is simple: If you barely know what you’re doing move on. If you have a firm grip on the details and a steady head, Gentoo can do everything and anything because it’s developers have created it to be a hackers delight.

I still find it interesting how many people cry out “you don’t have to update that much!” and ‘RTFM!’. But when reading the manual, this seems to be found prominently:

Code Listing 14: Updating your entire system

# emerge –update –deep world

Since security updates also happen in packages you have not explicitly installed on your system (but that are pulled in as dependencies of other programs), it is recommended to run this command once in a while.

(emphasis mine) and when introducing emerge world,

To keep your system in perfect shape (and not to mention install the latest security updates) you need to update your system regularly.

Being a Gentoo user for about 6 years, there’s no other OS that I’d rather install on a server. Currently at work we run Gentoo on 4 production boxes. One of those servers is our main production server which everyone in our company uses for just about all aspects of our business.
If you’re good enough with using and understanding Gentoo, you won’t have any problems at all.
The above blog could be related to any OS like Windows, BSD, RH, etc by anyone who doesn’t know enough about the OS to handle it correctly.
The below line states that the author doesn’t know Gentoo as well as he should to even consider running it on a server.
“This is hard with Gentoo. Gentoo wants you to change a lot of stuff. It wants to be bleeding edge”
I do agree with the comments about the install process of Gentoo. But like they say, ‘if you can’t take the heat, then get out of the kitchen.’

Personally Gentoo is the only Linux I’d use on a server, while it takes time to get installed correctly and I need a staging server to test major changes on, I feel it is the single most secure OS simple because I install what I need and nothing else, and other Linux install wans to and requires crap (sound anyone, sound on a server what the hell). I’m reminded of a version of Suse (old version of Suse) that wanted a menstruation calender as part of the server install (explain that). Gentoo is for the truly savvy installer only, but it is the only of for a truly minimal yet fully functional server install (think MC or screen, most servers do install them by default).

I have an older internal pre-production gentoo server that I “just” want to add odbc to php so I can connect to an ms sql database.

*Update USE flags to add in odbc…

~ # emerge -u php
Calculating dependencies… done!

>>> Emerging (1 of 66) sys-libs/zlib-1.2.3-r1 to /

There are important web apps running on the server that can not afford downtime. 66 packages = 95% chance something will go askew. I am now looking at getting a new server to install the latest version of gentoo so that I can update php (which seems a bit onerous to me)

It is a misconception that when Gentoo releases a new profile that you have to re-install Gentoo. Profiles typically involve things like a change in gcc version or some other significant software update.
I have moved through all my profile updates with very few problems.
People mistakenly think a new profile is like a new release of Redhat and do a re-install.
I have had Gentoo running for 3 years since my first install and am still ‘bleeding edge’ – try doing that with any other system .
Also, whilst Gentoo may take time to setup, it allows you to *only* install the apps you need, you don’t get all the superfluous cruft – you want an apache server? Who needs X, etc.
Also, you don’t *need* to do an ‘emerge –update world’ – although, imho, no harm if you have installed only the minimum packages. You can simply do a ‘emerge –update –deep apache’ and ignore the rest of the system.
Anyway, I can understand the time issue and certainly can’t argue with that.

One approach in gentoo that’s important to maintain any stability is to identify all the packages which should be kept cutting-edge, and which ones to only update when explicitly told to.

It’s very easy to just mask out all the packages you plan to only upgrade by hand. I do this to keep all the high-importance packages from being included in world emerges (eg networking tools, java). It’s handy to have a system that frequently updates low-risk applications. I get tired of administrating active servers where I’m stuck with tools that haven’t been upgraded since the turn of the century..

Nice article, Mr Ljungberg, I must say I agree with most of your points and long ago came to the same conclusion – Gentoo for fun and hacking and FreeBSD for serious professional server stability – so thanks for promulgating this. I don’t run servers these days so I don’t get as much runtime on FreeBSD as I would like but I run a Gentoo desktop at home and Ubuntu at work as it ‘just works’ and involves me in very little downtime.

Having messed about with UNIX professionally for >25 years and Linux >14 years I’ve got my grey stuff and would like to offer an alternative perspective – “judge your UNIX by the company it keeps”. If you want a serious, helpful, earnest community concerned with doing things right, then I would say FreeBSD is the place to be. The greybeards there have really got it right.

If you want the latest, innovative ‘stuff’ on a system that will force you to learn all sorts of nitty gritty details under the hood
then Gentoo is great. Just be willing to have to dig through huge forums for information – sometimes even for important release information which is sometimes hidden in obscure forum posts. Be ready for bad spelling, bad grammar, bad attitudes and often aggressive nastiness – as well as the huge volume of helpful information.

I think this is borne out in this blog too – just note from which direction the immaturity and name-calling comes!!

There are two distinct commentaries here that are actually useful and true.

1) You would have to be a bit of a fool (or uninformed, same difference) to use gentoo directly on a server. I.e. you should never be updating directly from the world at large onto the server, you should use staging servers. This requires extra hardware, and a good bit of time and effort, which leads to point (2)

2) gentoo can make a good server operating system, if you are willing and able to put in enough time. Unfortunately, this basically requires you to do the last stage of QA yourself, which most people don’t want to do (I certainly don’t). If you have a sufficiently large number of servers, this becomes a better deal.

Alexander, you are certainly entitled to your opinion, but it’s a shame that you choose to publish your ignorance to the world. The real title of your story should be “Why Alexander Shouldn’t Be Your System Administrator”. Unfortunately, your article highligts your shortcomings and short-sightedness as a systems administrator.

As other commenters pointed out, you don’t upgrade any production system without first testing the upgrade in a test/sandbox environment. This has nothing to do with gentoo, it’s just a common sense approach to responsible system administration.

Gentoo gives you tools that are unmatched in the industry (open source and commercial) for building a secure, well tuned, and up-to-date linux environment for desktops and SERVERS.

The gentoo toolset combined with a few configuration management tools (cfengine, subversion, chrooted build/test environments, and a few of your own scripts) gives you the ability to sucessfully administer a production environment of a few, or a few thousand linux systems.

In your defense, it does require more time to update gentoo than most other distributions. However, if the time is spent wisely, production impact is minimized and hundreds of new systems can be installed and configured in a couple minutes.

If you did a little research, you would be surprised to find number and diversity of people and businesses using gentoo for production servers and desktops.

I know my first paragraph is a bit harsh, but spewed ignorance does not sit well with me. Perhaps the title should be “Gentoo, A Great Linux Server, If You Take The Time”.

I am not a gentoo/linux zealot. I use Solaris for data servers and OpenBSD for perimeter firewalls. I simply took the time to understand how flexible and WONDERFULLY EXTENSIBLE gentoo really is.

As always, it boils down to a matter of choice. You are free to use whatever you want.

A fair number of commenters have expressed a similar opinion. What the argument really boils down to is, “I have two servers so it’s okay if one of them breaks once in a while.”

I would rather prefer having two servers where neither breaks ‘once in a while’. Your situation might be different.

Either way the time argument stands. You write that Gentoo is great ‘if you take the time’. As you’ll notice if you read my article, one of the reasons Gentoo was not optimal in my case was just because it took so much time to maintain.

If you have a different use scenario where uptime and maintenance cost is not important, please feel free to write your own article. But the intended audience of your article would not be the same as for my article.

I agree with you that any/all servers should not break, even the “build” server(s) used to compile and configure the updates.

I fully understand that the excessive time you spent on updating gentoo is unacceptable. It will always be unacceptable if done in this fashion for each machine.

When I say “if you take the time”, I mean take the time up front to create an automated “configuration management” structure around portage that will greatly reduce the administrator time needed for future updates. In other words, take time now to assemble tools that will later let you work smarter, not harder.

The bottom line is absolute minimum downtime and no “broken” time for production systems. The longest period of downtime is a system reboot.

I’m not trying to force you to change your mind, that would make me a fanatic. I’m trying to make it clear to your readers that there is real future value and time savings to be had by spending a little more time up front. The configuration management structure I built also works for my Solaris and OpenBSD systems. I could “plug” in almost any unix/linux variant. The time saved over the years has far out-weighed the time and effort spent “up-front”.

Somehow I get the feeling that we’ve had very dissimilar sys admin experiences.

Now, in your defense, I understand every single bit of your frustration with gentoo system administration because I, like everyone else who has worked with gentoo, have experienced the exact same issues and problems, such as: “Hey!!! I did exactly what the handbook told me to do and now my system is completely broken.” or “Oh crap, I’ve updated gcc and now I can’t compile anything” or “My emerge -e world bombed out after 273 packages, now my system doesn’t work right”. When this kind of stuff happens you either quit and say “gentoo sucks”, or you push on. You are perfectly correct if you quit at this point because it takes too much time and frustration.

When you decide to push on, you start reaping the real benefits. You start thinking about different ways of using portage’s tools, different ways of managing updates, and using other open source tools until you finally end up with a nice configuration management system that handles all your OS deployments and updates. Gentoo can teach you much more than how to handle frustration.

I think that everyone who says they love gentoo has also cursed gentoo at some point in the past (and maybe some point in the future too, because shit happens).

I have two objections:
In your article, you said
“1. Gentoo is (even more)Time Consuming to Install”
You don’t have to perform installation and maintenance tasks at this system. You can very well chroot from another faster machine to your Gentoo partition (even possibly over NFS), and do the installation/maintenance with that fast system.

And you said:
“3. Gentoo’s Stability Strategy: Update Everything”
YOU DON’T HAVE TO UPDATE. If in the future you want to update an installed package or install a recent package, just sync portage and if that package requires newer versions of libs installed in your system, well what would you expect? They would be updated as well. If you don’t want this to happen, NEVER SYNC and nothing in your base system will ever be updated; since in that frozen portage ‘snapshot’, the version of your required app. can only depend on those versions of libraries.

And don’t think that switching profile from, say, 2006.1/server to 2007.0/server will not result in sooo dramatical effects on your system, and everything would be broken. As I said above, you want to sync, update profiles (optional) ONLY IF you want the most recent version of an application.