Thursday, July 29, 2010

I had a "interesting" experience recently, namely had a EVA 4400 overheating due to environmental issues (fancy-talk for Aircon failure). The client phoned me, complaining that half of their Hyper-V VM's are not running. Further investigation revealed that the CSV's were offline. Hmmm, this was getting serious. I logged into Command View and saw that most of my VDisks were faulted, this was due in no small matter to the fact that all the drives in one of my shelves were faulted.

Event Logs

I had a look at the relevant EVA logs and discovered the following relevant entries:

In retrospect it was a fairly simple sequence of events, as evidenced by the entries above. The Air Conditioner failed, which caused the temperature within the Drive Shelf to rise (this is the HSV300 controller referred to in the event log). To prevent damage to itself, the drive then switched itself off, which prompted the log entry about the physical drive disappearing.

We then started seeing volumes transitioning to the missing state, i.e. our VDisks went missing. Hardly surprising considering that the drives containing them switched themselves off.

Resolution

Restored Air Conditioning (goes without saying I guess)

Powered off the EVA and all attached disk shelves

Powered on disk shelves and waited for the Numeric ID LED's at the back to display the proper IDs.

Powered up the Controller

Lo and behold! All the previously failed physical disks came on-line, meaning that my missing VDisks also made a most welcome return

Unfortunately my Hyper-V Hosts still couldn't access the Vdisks, so I had to unpresent and re-present them via Command View. I assume the EVA assigned new WWN's to the LUNs.

I re-scanned for storage from the Disk Management MMC on the Hyper-V Hosts

Brought the Disks and CSV's online via cluster manager

Started up the VM's

Conclusion

This was quite a harrowing experience, obviously. What struck me as ridiculous is that HP does not have *ANY* thermal shutdown logic / capabilities on the EVA controller itself. It keeps on trucking till the drives themselves fail, causing a very ungraceful failure of the VDisks. There is also no guarantee that your drives and VDisks will come back online. In essence - if your EVA overheats there is a distinct possibility that you lose your Data. Caveat Emptor...

Friday, July 2, 2010

Namibia is a third world....errr....developing country. So apart from rampant (55%) unemployment it also means that we are bandwidth starved. This also means that some businesses are running on capped accounts, so attempts need to be made to conserve bandwidth. I've had to move WSUS servers to new hardware or VM's a couple of times, and needless to say it's a huge time-sink and waste of bandwidth to re-download all updates every time you move your WSUS server. Thus I've come up with a way to move WSUS without downloading tons of patches - steps are outlined below for your enjoyment.

Install WSUS on your new server, making sure to select the option to use the existing Windows Internal Database

During the Choose Upstream Server of the configuration wizard, be sure to select Synchronise from another Windows Server Update Services Server

Ensure that the This is a replica of the upstream server check box is selected. This ensures that existing approvals, settings, computers and groups are maintained

Complete this Wizard

Your new replica server will synchronise with your upstream server. This is what we're talking about - no re-downloading many GB's worth of patches! N.B. Wait for this process to complete before carrying on with step 6.

Open up a CMD Prompt and navigate to "C:\Program Files\Update Services 3.0 API Samples and Tools\WsusMigrate\WsusMigrationExport" folder on your old WSUS server

Run "wsusmigrationexport.exe WSUS_Settings.xml" to export the settings. This will backup your approvals and target groups to an XML file

Transfer the WSUS_Settings.xml created above to your new WSUS server

Again navigate to "C:\Program Files\Update Services 3.0 API Samples and Tools\WsusMigrate\WsusMigrationImport" folder (on the new WSUS server). Run "wsusmigrationimport.exe WSUS_Settings.xml All None"

Review and compare settings on your two WSUS servers, ensuring that they match each other

Update the relevant GPO's to ensure clients are pointing to the new WSUS server

One last point, I've noticed that if you are not using GPO's to assign your computers to Computer Groups all your clients will get stuck into the Unassigned Computers group. You'll have to manually sort them into the appropriate groups again. Bummer, but you really should be using Computer Groups....

About Me

About This Blog

This blog serves 2 purposes. Firstly, I want to share information with other IT pros about the technologies we work with and how to solve problems we often face. I work with technologies from the desktop to the data center, Active Directory, System Center, Exchange, Hyper-V, VMware, Networking and Storage.

Less altruistically, I use my blog as a reference. There's so much to learn and remember in our field that it's impossible to keep up. By blogging, I have a notebook that I can access from anywhere. It has made me look much smarter than I probably am on many occasions.