Intro:
It all started in Dec '05. I arrived at my new job at a new startup as their jack of all trades IT guy for http://jumpcut.com/ They were running on three dev boxes with CentOS and having a number of samba issues, cvs, and apache issues. I'd been running Gentoo for about three and half years at this point and felt it could do everything we wanted and make my life easier. As the only sys admin, network engineer, desktop support, etc guy I took a vote and Gentoo won.

I: Fix the office stuff
Consolidated dev into one CentOS box. Installed Gentoo onto sys01 which was now free to be CVS with cvsd, which wasn't in the Centos package, Bugzilla, Cacti, net-snmp, RT, Nagios, Samba 3.0.22c or whatever fixed what Centos' Samba 3.0.14 couldn't do with the latests XP patches. Additionally followed the nice how-to the Gentoo Wiki to integrate Samba into the existing active directory.

Number of packages I didn't have to build RPMs for: 150+ mostly because of RT and Bugzilla deps.
Total time: two weeks for building, configuring packages, testing, and releasing to all employees

II: Moving more to Gentoo
Dev had standardized on PHP and then used a home built version of PHP 5.1.1 and Mysql 5.0.11 since neither existed in our CentOS. Additionally their original consultant did not setup software RAID in any of the dev machines. It was a bit of a mess, but we were early in the dev cycle so weird dependencies on custom builds had not crept in.

Built dev01 with LVM and software RAID, Apache 2.0, PHP 5.1.2 w/hardened patch and limited subset of functions to reduce binary size, current cvs ffmpeg and assorted libs, etc. Moving devs over and set each up with their own Apache instance. No one noticed a difference other than they really liked the default Gentoo colors in their terminal.

Number of packages I didn't have to build RPMs for: 50+ mostly due to ffmpeg and PHP required libs.
Total time: one week to build, configure, test, and release.

Number of packages I didn't have to build RPMs for: none
Total time: four hours as I used a stage4 I made from dev01

IV : Setting up production on Gentoo
We used what we called minimal stage4s generated from to dev to build the eventual twenty machines that ran the site. Each class of machine eventually got it's own stage4 like db-stage4, www-stage4, which allowed us to install a single machine in one hour or roughly six machines in one and a half hours which is the most we ever received at one time.
We rebuilt the database servers as x86_64 eight months later as well as the dev db server, but otherwise updated machines as packages became available. All new ebuilds were tested on dev, then staging, and then made it into production. Generally updates made it into production within three weeks of the initial package update hitting portage. Some hit portage quicker if they were security issues.

Total number of additional packages I never had to build: 30+
web stats, Postfix, postgrey are some of the highlights.
Total number of Gentoo caused problems in production: 0
Uptime: 99.99%, stupid power breaker in our colo.

V: How are we doing?
I've worked in a few large shops of 100+ machines and some small shops. As a one man shop I do far less work and update faster with Gentoo than I ever did with Redhat, Fedora, or Solaris in other < 50 machine shops. Remember as a startup being able to get new versions, especially versions that fix issues quickly and easily is key. Additionally using Gentoo ebuilds was vastly superior to any package I could build on my own. I don't have the time to QA in house packages and even if I did Gentoo builds are used by thousands of people rather than just one.
I don't come out so well against a large shop. However without the infrastructure or the budget to pay for large shop tools we did quite well. Additionally I am most comfortable in Gentoo and there is something to be said for using the tools you're most familiar with.
Where Gentoo has really shines is in projects that fail. No really. We've all done that "hey lets try upgrading to Apache 2.2 and see how well it works." In Gentoo you change a few lines, emerge apache, run some tests, realize it's not quite there, change a few lines, emerge apache again, and you're back to where you started. Total time about two hours.
Or even projects that go somewhere. "Hey I need X packages for testing." Gentoo installs, some minor tweaks, and hand off to the dev. When we go to production I know I can get the same package because I let Gentoo do the work rather than half ass a build because I didn't have time for non production issues when the project had no priority. Naturally there some changes in config in production, but we can go to production faster without having to repackage, re QA, and then release.

VI: Goodbye Gentoo servers
As of last week we are now owned by Yahoo! which uses Redhat. We're starting the process of moving to their infrastructure and it looks to be straightforward. I'm sad to see our Gentoo servers go, but trying to convince Yahoo! to run a distro they have no experience with just for us isn't going to happen.
I did however get geek points from a number of Yahoo engineers who thought it was very cool that we used Gentoo to build our system. Of course I never said things like, "Gentoo rulez" when I describe our infrastructure and I did say "We were relying on a number of bleeding edge packages so we picked a distro that catered to that. There are some quirks, but if you're prepared to deal with that it can be great."

It's sad to hear that yahoo is insisting on RedHat. I would not expect a conservative attitude from such a big company. On the other side, it makes the work of the IT Dep. much easier._________________Antonino Catinello | http://catinello.eu

The problem with large companies is that most decisions are made on the top and is unchangable. When you move to a structure like that it hurts performance on the lower levels, because the leaves hate to be dragged in the wind. The larger the structure, the more chance for resistance, expecially when leaves are transplanted.

Gentoo Servers!!!! I love gentoo; The server I set up at my high school (im now in college) is nearing its one year mark, as in uptime and age

Red hat is nice when you have the tech support, let them fix it when you dont know something. Of course this leads to lazy admins, but then again, its better then lazy windows admins._________________My Systems - "I suggest the whole thing be coded in whitespace. Henceforth the code will be obscure and functional at the same time."

I hate to say but personally I feel that Gentoo will not easily make it into the Blue Chip arena until such time one of the large vendors ie. HP, Dell provide commercial support for it. RedHat and Suse have the commercial ties, and upper management look far more favourable on these distros because of that. Commercial acceptance and support does not equal a better product though

Happy to see you had good time with gentoo.
I've also apreciated your frequent answers on the server mailing list, hope that you decide to keep at least some gentoo box and still lurk on our mail list.
Having prepared ppl like you help us is a great bonus to the community.

I'm in my final year of university (down in the land of Australia), and I've been using Gentoo as my distro for a few years. I even got a guy that I did group projects with into it. In our final year this year, myself, my Gentoo friend and some other guys had to do a full year project for a client external to university. Turns out our project wasn't going to be the standard software development stuff that they usually throw at you, to see if you've being paying attention the last few years. The project was more a research project, which means we needed a server box that was highly configurable and easy to rollback in case something went wrong. So my friend and I turned out to be the main code monkeys. Where did we turn - Gentoo of course

Turns out my old man was upgrading, so I got his old machine, installed Gentoo within a couple of days, whacked in on the back of the uni network, and we were off. Subversion, apache2, and all the Java tools (being the language of the project) were all installed quickly and efficiently. When we wanted to try something, we let portage do the work, had a play and if it didn't work, a couple of hours to undo the damage. If it did work, it stayed. In terms of software configuration the biggest hurdle was documentation of some of the software, but that's not Gentoo's fault.

All in all the group agreed that if it wasn't for Gentoo, we would have never gotten the project done in the allocated time span. Thanks to portage, we didn't have to spend huge amounts of time dealing with dependencies, or getting the damn thing compiled. No we sat back while Gentoo did it's thing, and got the important stuff done.

Doesnt portage have an option to grab precompiled packages? It would be interesting if someone were to start a commercialized branch of gentoo that had the option to use bin packages for most of the packages (or atleast a system to drop request for precompiled).

I dont mind the other distros, but I love gentoo, it doesnt complain, it works great, and I think businesses would love it IF they were to know about it.

I only figured out about gentoo because I heard about it from the 3rd person from one of my friends. Maybe we should start a gentoo awarness initiative?_________________My Systems - "I suggest the whole thing be coded in whitespace. Henceforth the code will be obscure and functional at the same time."

I figured it was about time Gentoo Chat had a good sucess story rather than some of those *other* posts.

In regards to Yahoo I still remain the Jumpcut Tech Guy or whatever my title ends up being, but Servers, Storage, Networking, etc are all individual groups outside of us at Y! They install and manage the servers that us, Flickr, de.lic.io.us, Yahoo Mail, etc all use. Trying to add Gentoo to that infrastructure would be impossible since they're already managing thousands of RedHat servers. However a few Y! engineers expressed intrest in setting up a few boxes to play with.

Of course some of the local boxes in the office won't be changing like sys01 and my workstation. I'll still be on the mailing lists and hanging around in the forums. I also keep meaning to sign up for x86 and amd64 arch testing, but have never had the time as you might imagine. My personal webserver will also stay Gentoo so I'm not going anywhere, just biding me time until I can Gentoo-ize another shop.

most people dont even know what compiling is... people fear the unknown..._________________My Systems - "I suggest the whole thing be coded in whitespace. Henceforth the code will be obscure and functional at the same time."

kashani, sir, you really should set a page, a blog, or a very complete guide about how to manage 100+ Gentoo machines.

At my last job, I actually did manage close to 1,000 servers all running Gentoo. It was no picnic, but I've since realized that there really are few tools out there in any distro to manage a huge enterprise like that. I'm at a much smaller shop now that runs CentOS, and the problems of maintaining the few dozen machines we have are really not very different from maintaining 1,000.

With Gentoo, you really have to treat it as a meta-distribution and, at least to some extent, create your own distro (or at least image of binary packages) to roll out; i.e., unless you want to devote all your machines to being a distcc cluster, compiling every app for every machine just isn't practical. The really big problem, one which I still haven't completely solved, is that of configuration file management. Running etc-update or dispatch-conf on 1,000 machines is not only impractical; it's impossible. Software solutions to synchronize config files across a network become very attractive.

What really keeps me with Gentoo--and the reason I'm going to push to migrate to Gentoo at my new job--is that it's elegant. Even if Gentoo was not a meta-distribution with installation from source, I would still be in love with its rc initialization system (with dependencies!!!) and elegant management scripts. CentOS just can't compare in this regard.

Whenever someone asks me how to make an init script I always ask which distro they are running, if its gentoo ill gladly help them, but red hat needs to despirately do something with their startup, its a royal pain without a gui_________________My Systems - "I suggest the whole thing be coded in whitespace. Henceforth the code will be obscure and functional at the same time."

y'know, it's been just great reading this. Sometimes I was close to tears from sensing the passion you guys have for gentoo. And do you know what's the greatest thing about that is? I can relate! I've got only one gentoo box (started Jan 02, 06) and I love it! It may seem somewhat geekish to love a (meta-) distribution the way we love gentoo, but, hey, we do love it. Both portage and the community have become the main reasons for me turning into a linux missionary more than ever.

I'm losing my point. What did I want to say? Basically thanks, kashani, for sharing, and also thanks whoever got this thread mentioned in the gwn._________________

Quote:

This time it was right, it would work, and no one would have to get nailed to anything.

Sadly, however, before she could get to a phone to tell anyone about it, a terribly stupid catastrophe occurred, and the idea was lost forever.

The really big problem, one which I still haven't completely solved, is that of configuration file management. Running etc-update or dispatch-conf on 1,000 machines is not only impractical; it's impossible. Software solutions to synchronize config files across a network become very attractive.

Have you considered cfengine for those tasks?_________________Please add [solved] to the initial post's subject line if you feel your problem is resolved. Help answer the unanswered

VI: Goodbye Gentoo servers
As of last week we are now owned by Yahoo! which uses Redhat. We're starting the process of moving to their infrastructure and it looks to be straightforward. I'm sad to see our Gentoo servers go, but trying to convince Yahoo! to run a distro they have no experience with just for us isn't going to happen.
I did however get geek points from a number of Yahoo engineers who thought it was very cool that we used Gentoo to build our system. Of course I never said things like, "Gentoo rulez" when I describe our infrastructure and I did say "We were relying on a number of bleeding edge packages so we picked a distro that catered to that. There are some quirks, but if you're prepared to deal with that it can be great."

I had a similar experience working for an ISP where I eventually standardized on Gentoo for almost all Linux boxes - its a lot less hassle as far as maintenance goes than RedHat. I just started a new developer job about 3 weeks ago (so Im no longer doing sys. admin. work). This is after having worked for the ISP for 4.5 years as sys. admin. Im not assuming my replacement will want to continue working with Gentoo (though, our network guy, who's a FreeBSD nut, loves our Gentoo web servers).

BTW, instead of saying "Gentoo Rulez" you could have pointed out a few high profile projects that *do* use Gentoo (e.g. Asterisk).

The really big problem, one which I still haven't completely solved, is that of configuration file management. Running etc-update or dispatch-conf on 1,000 machines is not only impractical; it's impossible. Software solutions to synchronize config files across a network become very attractive.