Property management at my organization has informed me that our building will be losing power for 4 hours tomorrow. I need to be prepared for this event (we're a small organization, i'm young, therefore I am I.T). What sorts of things do I need to be aware of.

I am planning on going in and shutting down all machines and printers. Will this cover me? We have a managed switch. Does it need to be shut down? Do I need to disconnect plugs in case of a surge?

Seems like I'll be covered all around if I just unplug everything. Thanks for any insight though.

7 Answers
7

Power everything off - workstations, servers, printers, switches, the works.

Turn off your UPS' so they don't panic when power is lost.

After outage in this order:

Turn on UPS

Turn on networking (router, switches etc)

Turn on servers

Turn on workstations

Turn on everything else

Have a test plan ready so you can test important functionality is working

Internet connectivity

Email, printing etc

If possible, have a laptop with a separate network connection handy (ie: you can get to the internet without your work router working). That way you have a way to ask for help here if something goes wrong with the networking when it comes back up. :)

You should be fine though - the fact that you took the time to ask here shows you already have the requisite "clue" required for IT support!

+1 for the laptop with its own internet access. In addition to calling for help, you can use this to test that outside items are available beyond your network and inside resources aren't. That was a part of your test plan right? Riiight?
–
Rob AllenJun 6 '09 at 18:36

4

+1. A couple of things: if you have the login for the managed switch, make sure you log in and save the configuration. Also, see if you have the cables required to connect to the console port of both your router and switch. Many of them require serial (DB-9) connections, so if you have a recent laptop, you may need a USB->serial adapter. Good luck.
–
Murali SuriarJun 6 '09 at 18:43

2

+1 and also figured I'd repeat what some of the posters have already said below about unplugging some of your valuable hardware from the WALL if you don't know exactly what the facilities people are going to be doing to the power lines. Better safe than sorry!
–
KPWINCJun 7 '09 at 6:25

Yeah, definitely isolate as much from the supply as you can. However, don't have anything running on the UPS when it's isolated, as you NEED the ground connection if anything is online.
–
Chris ThorpeApr 24 '11 at 6:31

Having just done through a datacenter shutdown in the last week, this is fresh on my mind ;). Yes, shutting everything down needs to be done. Some things can tolerate having the power yanked out from underneath them, and they typically can be identified by not having a power switch on them. Depending on what the heck the facilities people are doing, you may want to physically unplug higher value assets from the wall.

Sometimes you can get sizeable power-surges when power returns, and that can kill gear. Also, be aware of your start-up load. EVERYTHING powering on at once may be enough to pop breakers and bring everything down again. That sucks. Try for a staged power-on to get around this.

EDIT: We had a case where the transfer switch for the generator didn't fire, and the UPS ran out of batteries. When utility came back, everything powered on at once. 20 seconds later the UPS hit Overload and dropped load from the room again. By then enough techs had arrived on site (it HAD to happen during a weekend) that they managed to get enough stuff turned off at the rack-switches that when utility came back again the room didn't drop again. That yo-yo cost us a lot of hardware, and it was week recovering from some of the damage.

Definitely agree- make sure that you turn on the servers one at a time, and let them get the disks fully spinning before you bring up the next one. Otherwise, you can easily pop the breaker- I've done it myself.
–
Tim HowlandJun 7 '09 at 3:32

The only thing I'd add is to make sure you have all your support contracts and replacement parts easily findable. I'm sure you do, but double and triple check that. If you've had servers that haven't powered down in a long time you'll occasionally find issues with all the moving parts (hard drive bearings/motors, fan motors and bearings and some really silly stuff that we normally don't think about).
If you can be there for the duration, make up a list for your important servers including their service tag, configurations, where the backups are and any people that might be responsible for apps on each machine. Tape that up on the side of the server in question, or have it in a binder in the datacenter for when you start trying to bring things back up. You don't want to have to keep running all over your office if something is broken.

First of all, you've got good backups, right? :) Always a good idea to make sure that you're all backed up, just in case a disk decides to not spin up, etc. when you bring everything back online. Nothing like breathing easy after a power outage then realizing that one critical SQL server never came back. :)

Second, you're right, gracefully shut down and turn off all servers, etc.

If it's not a modern building and/or you don't have a UPS in-line with surge protection, (which it sounds like) or anything like that, it's always better being safe rather than sorry, and I would suggest unplugging equipment. Bringing power back online could result in a power spike.

Also, a little FYI; bring your core servers down last, (such as DNS, etc.) You don't want to bring down your DNS servers and find yourself unable to resolve the rest of the servers if you're shutting them off remotely :)

Shut everything down, including switches and routers, and either unplug or turn off at the wall socket.

If you're worried about surges, is there a budget to get a few small UPS to cover the most critical equipment?

EDIT:
I like neobyte's suggestion to have a "test plan" for when you bring stuff back up. Even if you just have one in your head, and a firm order to follow to restore services quickly you'll be fine. Make sure people know that you're working to a plan and hopefully they'll understand why switching on their workstation before you've looked at the servers and switch isn't going to help!

Inter-server dependencies / server boot order will be best laid out ahead of time, too. For example, if it's a windows network, power up DC's first, then database servers, then exchange, etc. This is a great list for your documentation, in any case.