Are newer rack servers able to protect themselves better than the ones in this room?

Several hundred 50-cent temperature sensors and power cutoff switches embedded in various things (power supplies, boards, disk drives) have got to cost less than a tertiary a/c.

No, I know you people just purchase whatever is available, and have little if any say in the parts that go into the equipment. But by now, things have got to be perfected to the point where you can add a server farm to your shopping cart and have it arrive next day.

With all the power that all those big computers in that room use
Gee...I wonder what is causing all the heat that is making these AC units run constantly
I doubt it is really eco-friendly to run those big units at the same time.
I'm a liberal! I'm smucking fart!
And all computers have cooling fans anyway, so why put the A/C for the building in that room?
I need to point this out for all the dumb power-hungry conservatives in our IT dept
In the future, let's try to be a little more conscientious of our energy usage!
I just saved $4 dollars' worth of electricity and it only cost us about 200 grand in equipment!

I know you people just purchase whatever is available, and have little if any say in the parts that go into the equipment. But by now, things have got to be perfected to the point where you can add a server farm to your shopping cart and have it arrive next day.

Man I wish I lived in your world. Around these parts, your new domain controller is the the HP Pavilion from 4 years ago that the boss's kid doesn't want anymore.

interesting.. that showed up as a posting on the livejournal techsupport community right after memorial day weekend from someone who claimed to work at the office where it happened. the post no longer exists.

No, I know you people just purchase whatever is available, and have little if any say in the parts that go into the equipment. But by now, things have got to be perfected to the point where you can add a server farm to your shopping cart and have it arrive next day.

It also depends what vendor. If you order a Sun server with 4 GB of memory and add 4GB more, they put it in another cardboard box and ship it along with the server.

The Facility Manager should be fired for leaving his keycard somewhere where an unauthorized person could access it and so gain access to several hundred thousand dollars worth of equipment, which he proceeded to "accidently" destroy. If he couldn't access the server room, he wouldn't have been able to shut down the AC.
There is also the backup--have the computers send your cell phone a text message (or beep you in earlier times) when they start to fail. ("Oh my god, I can't talk to ExServ123! ExServ123 just died! Someone help, I just lost ExServ345! Gack... byuuu....") I've heard people doing this for all sorts of technical problems.

The idea that that many critical servers would not have some sort of capability to signal someone if they failed is ludicrous. Is the IT team totally dependent on user complaints to find out when a server goes down during working hours?

Just wait until those high-quality 50-cent temperature sensors start shutting down your domain controller and it takes you three or four hours to figure out that the 50 cent temperature sensor is bad and has kept you from doing any real work because some tight ass couldn't fathom spending money on a tertiary A/C.

The idea that that many critical servers would not have some sort of capability to signal someone if they failed is ludicrous. Is the IT team totally dependent on user complaints to find out when a server goes down during working hours?

This is a really good point, at the least there should be some sort of automated email system for the temperature of the room, I couldn't imagine the cost being all that high.

The bigger issue as I see it is either:
a) ----- should have access to the room. If this is the case, why did you hire someone that would make that big of a mistake so easily to do a job where they'd need that kind of access?
b) ----- should NOT have access to the room. In this case, why is there no lock on the server room door to keep such people out?

Not so sure this is fake. Yes, we've all heard the urban legends, but I've gotten more than one page from my systems telling me that this or that alarm has triggered (along with some relevant critical numbers). Invariably, it's because some idiot did what s/he wasn't supposed to do.

Many years ago, I had built a simple web-entry system (the department had us use pure html, cold fusion and a sybase DB). Like most folks, I created a script to create and initialize the DB. In the script, I followed the typical: drop, then re-create pattern. When everything was working, I changed the script to print out a big "Do not run this as it will destroy the DB" message (at the top) and then exit, before all the actual DB stuff. For extra safety, I made the script read-only.

A few months later, I was put on another project, and some new guy was brought in to maintain the system. One day, my old boss comes over to me and asks me to help them figure out why the system seems to have lost much of its data. I go to enter one ticket and notice the sequence numbers have been reset to the starting value. I ask the obvious question, and the guy tells me that he saw the script was read only, so he made it executable. He saw the message, and commented it (and the 'exit') out. Why? Because he wanted to see what it did. headdesk

I assume this was in reference to my post, since I used the word "liberal" (you know there is a quote button, right?)

I in no way inferred anything about "all" liberals. I merely attempted to inject humor into a thought process that was obviously incomplete in its logic.
True, I made an assumption that "eco-friendly" was more likely to be used by a liberal than a conservative...call it poetic license and chill out.
Luckily I don't program anything, I am a network admin. So don't worry about my programming skills.
I am smart enough to know that if AC units are constantly running, you might want to figure out what is causing the heat before shutting them off.

People should pass little intelligence tests before having access to keys to such places though.

Remember, this is a state agency. These are Government employees.
It would be a violation of (some sort of) civil rights to test intelligence; as would be the case with competence.
{And, probably, cause a Union hubub too}

My former boss did something similar in a client's datacenter. Unfortunately he should have known better and didn't even have the eco-excuse, he just got cold while working in there and forgot to turn them on again before he left.

Could be. Considering I'm a part time admin with no pager and no cell phone and no way to tap into the phone system anyway, I don't have anything like this.

There are four other people actively involved with our servers, two of whom do it full time. A few weeks back a water leak blew up the main power distribution panel feeding the server room. That caused the generator to kick on. Everything at that point was fine, but no one was notified. Then, at about 2 a.m., a hose blew off the generator, which promptly shut down. Again, no notifications went out. One of the full time IT people got a call as he got out of the shower at 7:30, "Hey, did you know everything is down?"

So, yes, the IT team could be totally dependant on user complaints. That sort of thing costs money. And if it's a little too much, it doesn't get added.

This reminds me of when I worked at a state agency (different state) and they were trying to figure out how to save money. They decided to cut power to the building for the weekend. But they forgot to tell the IT department about it. So when everyone came in Monday morning, we couldn't get any work done until about noon when they got the network back up.

b) ----- should NOT have access to the room. In this case, why is there no lock on the server room door to keep such people out?

Read the letter again: it WAS locked, but ----- went and stole the key from [the facility manager’s] desk. This is why I say [the facility manager] should be fired, leaving a key where an unauthorized person can access it is worse than turning off 3 AC's.

Reminds me of when they got the thermostats reversed for the server room & the programmers room. As the servers (AS400's) warmed up it got colder in our room, which kicked on the heat in the server room... It was around 120-130 degrees when someone finally went in (because a PC-based print server died). The 400s never even hiccuped.

Could be. Considering I'm a part time admin with no pager and no cell phone and no way to tap into the phone system anyway, I don't have anything like this.

There are four other people actively involved with our servers, two of whom do it full time. A few weeks back a water leak blew up the main power distribution panel feeding the server room. That caused the generator to kick on. Everything at that point was fine, but no one was notified. Then, at about 2 a.m., a hose blew off the generator, which promptly shut down. Again, no notifications went out. One of the full time IT people got a call as he got out of the shower at 7:30, "Hey, did you know everything is down?"

So, yes, the IT team could be totally dependant on user complaints. That sort of thing costs money. And if it's a little too much, it doesn't get added.

I really wonder when people don't stop and think "someone must have come up with a solution for this?". You know how 'much' server monitoring costs? NOTHING! It's called Free and Open Source Software, in this case, the excellent Nagios.

Could be. Considering I'm a part time admin with no pager and no cell phone and no way to tap into the phone system anyway, I don't have anything like this.

There are four other people actively involved with our servers, two of whom do it full time. A few weeks back a water leak blew up the main power distribution panel feeding the server room. That caused the generator to kick on. Everything at that point was fine, but no one was notified. Then, at about 2 a.m., a hose blew off the generator, which promptly shut down. Again, no notifications went out. One of the full time IT people got a call as he got out of the shower at 7:30, "Hey, did you know everything is down?"

So, yes, the IT team could be totally dependant on user complaints. That sort of thing costs money. And if it's a little too much, it doesn't get added.

I really wonder when people don't stop and think "someone must have come up with a solution for this?". You know how 'much' server monitoring costs? NOTHING! It's called Free and Open Source Software, in this case, the excellent Nagios.

When I worked at Eaton in the early 80s, upper management all smoked. At least one of them regularly smoked in the computer room. As a junior programmer who didn't happen to smoke, I asked him if that was such a good idea because the (300 megabyte washing machine) disk drives ran at such tight tolerances. There was a poster at the time that showed that disk heads 'flew' lower than a human hair :) :) His reply was mostly non-verbal -- he dropped his cigarette on the raised flooring next to the closest disk drive and commented disdainfully that "Disk drives have HEPA filters." Something less than a month later we had a disk drive failure compounded by a less than knowledgeable operator taking the trashed disk and trying it in most of the other drives before he was stopped. -- After the massive cleanup, a no smoking rule was posted. I wasn't senior (or ballsy) enough to ask the management individual how he felt about having contributed to the mess. The money didn't count -- we were working on a DOD security project.

I vote for a fake. I don't doubt that one day an idiot would get up and do something totally stupid like turning off the AC in a server room. Well, actually, I believe that every day, an idiot gets up and does somthing that stupid.

But the e-mail - I don't believe it. Not only does this guy steal a key to get unauthorized access to the server room, in order to switch off the AC without authorization, but then he boasts about it on the company mailing list? And chastizes his coworkers for leaving the ACs running?

And the most suspicious thing: He destroys $200k worth of equipment and they merely lay him off instead of also suing the sh** out of him?

Then someone pointed out that similar but not pin-pointable versions of this story have been circulating for a while - this is very common for urban legends. I think this is the Republican version of the "Bush holds children's book upside down" or "Bush forgets to take caps off binoculars" legends, fabricated to make fun of $POLITICAL_OPPONENT, in this case "liberal tree huggers", obviously.

If the story were actually true, the Real WTF would be that in a server room stuffed with hundreds of thousands of dollars there is no temperature sensor that can trigger an automatic shutdown and service personnel alerts.

I vote for a fake. I don't doubt that one day an idiot would get up and do something totally stupid like turning off the AC in a server room. Well, actually, I believe that every day, an idiot gets up and does somthing that stupid.

But the e-mail - I don't believe it. Not only does this guy steal a key to get unauthorized access to the server room, in order to switch off the AC without authorization, but then he boasts about it on the company mailing list? And chastizes his coworkers for leaving the ACs running?

And the most suspicious thing: He destroys $200k worth of equipment and they merely lay him off instead of also suing the sh** out of him?

Then someone pointed out that similar but not pin-pointable versions of this story have been circulating for a while - this is very common for urban legends. I think this is the Republican version of the "Bush holds children's book upside down" or "Bush forgets to take caps off binoculars" legends, fabricated to make fun of $POLITICAL_OPPONENT, in this case "liberal tree huggers", obviously.

If the story were actually true, the Real WTF would be that in a server room stuffed with hundreds of thousands of dollars there is no temperature sensor that can trigger an automatic shutdown and service personnel alerts.

You must be fresh out of college or something. I've met enough self righeous idiots to totally believe this, and enough penny pinching managers to believe they'd leave out the monitoring and alarms.

People should pass little intelligence tests before having access to keys to such places though.

Remember, this is a state agency. These are Government employees.
It would be a violation of (some sort of) civil rights to test intelligence; as would be the case with competence.
{And, probably, cause a Union hubub too}

LIBERAL government employees must be mega stupid then?

In my super liberal country (New Zealand) government workers actually get paid more then the private sector. Sure governemnt is big and dumb, but the people who work there are mostly pretty on to it.

Well, at a former job, we had a machine that would send messages to a another machine that was offsite. If the offsite machine did not hear a message every 2 minutes or so, we'd get paged.

Of course, that cost money. The way I've seen governments work (thru colleges) is that if you can't directly attribute a cost of something to a project, it doesn't happen. So, it wouldn't surprise me if there were no resources for monitoring, as there would be no budget for it.