13 September, 2007

Datacenter Confidential #6

While the average systems administrator harbors disdain for a large number of professionals, including but not limited to marketing and/or public relations, sales, middle and senior management, she or he reserves a special kind of burning hatred for those freeloaders who call themselves "project managers."

The "project manager" is a dangerous beast, and it is incumbent on the systems administrator to deflect, devalue or otherwise avoid the project manager altogether.

Project managers are the natural enemy of systems administrators because they are skilled hucksters, con-men and women, do-nothing hangers-on sucking gleefully on the teate of the tech industry. What differentiates them from the other freeloaders, opportunists and ne'er-do-wells who are attracted to that good start-up dollar is that they get in the way of the well-established sysadmin racket.

Let me explain.

For a long time I struggled to come up with a succinct definition of what my job entailed until my hero and mentor, one chris f, spelled it out thusly:

Systems administration is nothing more than a protection racket, basically. The reason you got hired, the reason I hired you, was because you are a general Unix and networking handyman. Your primary function is to unfuck things when machines or routers or switches or firewalls crap out -- and they will. And when you set up a network or servers, you do so with the laziest, easiest fixes in mind. If you do it right, and you have enough money, then there is not much work for you to do. But the company continues to pay you because (1) no one else knows just how to fix everything and (2) they call on you at all hours, night or day, week or weekend, to fix things

Personally, I've always found this adage useful: programmers are too lazy to sysadmin, and sysadmins are too lazy to program.

Let's use a recent work event to illustrate the uselessness of the average project manager:

Our developers, and to some extent operations (meaning, basically, me) use a source code revision control system called "Perforce." Our perforce server was running on an MS Windows 2003 machine, a desktop computer with a couple of stand-alone disk-drives, a slow CPU and a meager compliment of RAM. The version of Perforce it was running was about 5 years old. I had mentioned to a number of people between the Fall of 2005 until I had given up in the Spring of 2007 that we might want to upgrade this, perhaps budget some money to find a better machine to replace the Windows desktop it was running on, etc.

My pleas and warnings were ignored until this Summer, when the fancy Project Manager suggested that we upgrade Perforce so we could use some Cool New Features. All of the sudden, upgrading Perforce became a Priority.

Alas, the upgrade was not so high a priority that there was a budget attached, and so the project waited until I was able to decommission a machine out of production from our datacenter and retro-fit it. This would have taken longer had the desktop running Perforce not started crashing at random intervals.

Soon, the project went from high to urgent priority. This happened conveniently a week before my short Labor-Day Vacation.

In a perfect world, a sysdamin has plenty of time to sit around, read Usenet (or, these days, salon.com and myspace.com), disparage her or his coworkers and still have time to turn the occasional adversary's home phone into a payphone.

In my world, I was being hit with customer case work, training my subordinates (or, the IT guy and my boss -- every sysadmin knows that a good boss is also a subordinate), new machine installs (many riddled with odd problems) and a larder of personal real-life issues that would make for a good TV drama.

In the fray, I think I agreed to upgrade Perforce the weekend after my return from a much-needed vacation in New York. I may have said that, but it came as a surprise to me when, on my return, my boss reminded me of it.

And pushing for this upgrade, making sure to CC the CTO on all his requests, was this fucking Project Manager.

I, of course, pushed back.

Normally my boss, the Director of Operations, would dutifully acquiesce to my request for more time. But this time there was a "my job is on the line with this one" tone to his voice and I knew I had to bite the bullet and "get it done."

As it turns out, my request for more time would have been the right move -- as easy as Perforce upgrades generally are, when you skip ahead 5 years and move from Windows to FreeBSD, several things pop up to bite you in the ass.

And ahead I went, there but for the grace of Perforce, upgrade woes be damned. As soon as things started to break, the delicate waltz of blame deflection began. Ultimately, the question of blame is very simple: a project manager insisted on a rushed upgrade, and, without ample time to regression test, I moved ahead on the upgrade under pressure from senior management. The myriad problems that resulted in this action were not a result of my lack of foresight, but the Project Manager's lack of foresight and the pressure on my boss from the CTO who was bedazzled by the Project Manager selling this project without fully considering the possible impact.

The problems with the upgrade were largely minor or cosmetic, yet, and here's another reason why I hate Project Managers, the onus was on me to fix these problems, and the perception was that I was at fault. In the end, two people fixed all of the known problems: me and a former developer (who fixed a bug in a java API that interfaced with Perforce).

I don't want a pity party, or a cake, or an award. I just want to be able to fuck off for a couple of weeks without too much being asked of me.