The machine is a Dell PowerEdge 2600 server running Windows Server 2008 trial 32bit (yah, its not supposed to...but it works! [well, it used to]).

For the sake of confusion: the drives are numbered 0, 1 and 2.

I was coding away as usual when I noticed the Dell logo on the front of the case was orange. So I opened the case door and saw that the HD vents were completely covered with dust (I know its not related to the orange light...but I hate dust). Since the drives are hot-swappable I yanked drive 2 out and cleaned off the dust and put it back in. I then yanked drive 1 out and cleaned the dust off that one and put it back in. Someone asked be to help set up a printer on their machine so I got up and 20 minutes later came back to see 'No boot device available - strike F1 to retry boot, F2 for setup utility` displayed on the server's monitor. I look down at the drives and drives 1 and 2 have orange lights instead of the green ones!

Reset the bios on the original PowerEdge 2600 (pulled the bios battery out). All drives appear fine. Get the Missing operating system message when booting. A drive lights are green.

Booted Ubuntu from a CD to inspect the drives. 2 of the drives
are displayed in Computer. Since the data is striped the files/folders in the drives are gibberish.

Booted Ubuntu and opened Terminal and executed sudo fdisk -l which listed the 3 drives. On the 3rd drive listed it states Disk identifier: 0x00000000 Disk /dev/sdb doesn't contain a valid partition table

Do you think the drives ARE actually toast?
Could it be SCSI or other hardware failure?
Could it be incorrect System Settings?
Is there any way to create a virtual RAID in Ubuntu on the 2 drives that are "valid" so I can copy the data to a network share?
Should I try reinstalling the Windows Server OS(eek!)?
Do you have any suggestions that I can try?

UPDATE

After doing lots of googling I came across Raid Reconstructor. I tried this program using my Dell PowerEdge 2600 using a bootable windows XP CD but it did not work (no drives detected). I then installed two of the drives into the PowerEdge 2500 alongside the 2500's existing single-drive RAID 0 running Microsoft Server 2003. I then installed and activated Raid Reconstructor which created a virtual image of the RAID-5 array, opened the image with Captain Nemo, and backed up my C:/Websites directory to another computer...with ALL files intact (so far)!!!

I will hopefully be able to restore the drives 100%

Lessons learned:

I don't care if the server can "hot-swap" drives. DON'T FREAKING DO IT!

Back up your data, dummy!

Thanks for all your help, answers, and comments (and for being wrong about the data loss. haha)!

Of course you can hot-swap drives with no problems, assuming you do it right. You can't just yank a drive, put it back, and immediately yank another drive. Arrays need to rebuild after a failure, and removing a drive is treated as a failure. Just because you didn't know this doesn't mean that there is a problem with hot-swapping drives.
–
MDMarraJun 4 '10 at 22:49

I've spoken to others who had bad experiences with hot-swapping drives in Dell x9xx machines. On Dell 2950 and 1950 machines, we've had a number of problems when hot-swapping a drive (both SAS and SATA drives). It should work, but I've seen some pretty bad failures, but for us most of the failures were recoverable.
–
Stefan LasiewskiJun 4 '10 at 22:57

4

@Stefan: The issue isn't with the hot swapping at all. The problem is when drive 2 was pulled, the array would have gone to Degraded state, then when drive 1 was pulled, the array would have gone Failed, and as Chopper3 aptly puts it - is toast.
–
Ben PilbrowJun 4 '10 at 23:02

@Stefan - We are almost exclusively a Dell shop and have dozens of 19xx and 29xx series servers in production and haven't had a single issue with hot-swap. We also keep the RAID controller firmware up to date when the release notes indicate a worthwhile fix. But many failed drives have been hot-swapped by me personally in those systems with no ill effects.
–
MDMarraJun 4 '10 at 23:08

3

A better clarification of, "I don't care if the server can "hot-swap" drives. DON'T FREAKING DO IT!" would be to make sure you thoroughly understand RAID, what it means, what the different RAID levels are, and how each RAID level handles disk loss/removal. If this has been a RAID10 array, you probably would have been fine (assuming you didn't pull two disks from the same mirrored pair). It all comes down to understanding your system before you touch things like that. Hot Swap was never the problem here. Ignorance was.
–
Christopher CashellJun 23 '10 at 16:21

3 Answers
3

The drives themselves won't be physically broken, this is simply that you've killed the array by removing a second disk before the first one had rebuilt - I'm >90% sure your array is toast. Basically you shouldn't have removed them at all while live, if you absolutely had to you should have waited for the array to rebuild before doing the second disk.

Sounds like this is exactly what happened.
–
egorgryJun 4 '10 at 14:20

Yah. Thats what I was afraid of too.
–
David MurdochJun 4 '10 at 14:26

1

Never, ever pull a working drive in a live server unless you absolutely have no choice in the matter (like someone has a gun to your head). Shutdown the server (or heck, yank the power cords) before ever messing with working hardware.
–
Chris SJun 4 '10 at 14:26

Hope you didn't have anything planned for the weekend :( - I think we've all found out the hard way that you should never remove a disk just for the hell of it - I did just the same kind of thing back when you were in kindergarden ;)
–
Chopper3Jun 4 '10 at 14:27

I still get nervous swapping drives on our netapp with raid-dp. :) Funny little story I once worked with a gentleman that tried to hot swap a scsi cable on a HPUX system, I think a K580? That didn't work out in his favor.
–
egorgryJun 4 '10 at 14:35