3ware 9500S RAID Controller

I have a file server running Server 2003 SP2 with a RAID 10 array on a 3Ware 9500S-12 card. Drives are Western Digital WD1002FBYS 1 Tb.
We have begun having severe disk performance issues with this machine and I cannot determine why. We have very bad fragmentation and we cannot defrag (hangs at about 3% no matter what software is used)
I would assume that I have at least one failing disk in the array, but the 3DM2 site shows healthy disks. The only indication I saw that there might be a bad disk was last night in the Drive section... one of the drives was highlighted in yellow. There were no alarms and I could not find any documentation on that that means (I assume its bad!). This morning, the highlight is gone.
The controller will give me a SMART readout, but it is all in hex and I cannot find any info online on how to interpret it.

All that said, has anyone encountered the yellow highlight before?
Does anyone know how to read the Western Digital Hex readout?
Is there anything else I can do that I am not already doing? At this point, I am tempted to start swapping out drives. 48% fragmentation is killing us!

You have unrecoverable read errors most likely. Run the windows chkdsk option with the flag to check & repair bad blocks for the entire volume. (Which means doing it at the next reboot).

Most likely a combination of unrecoverable read errors and filesystem corruption. If you are lucky, the check disk will be able to repair it. If not, then you'll have to create a full bootable backup, blow the RAID away, reinitialize it, then do a restore.

Checkdisk found and repaired a few errors. there were no bad sectors and no unrecoverable errors. Using Diskkeeper and a lot of patience I have finally managed to defrag the drive more or less but the read write performance is still terrible.
Now the 3ware driver has begun spitting out errors every hour into the application log.
Typical of 3ware, there is no reference to what they mean to be found anywhere online.
They are all event 3 and have entries like:
Packet: Id=80 Opcode=0x10
Sense: Stat=0x2 Err=0x10D Slen=18
MODE_SENSE6 Unit=0 Len=64
FW: INVALID_OPCODE (opcode=0x4D)
Packet: Id=114 Opcode=0x10
Sense: Stat=0x2 Err=0x101 Slen=18
0x4D Unit=0 Len=4
etc
They arrive in one big block, once per hour, so clearly it is the result of some kind of scan the controller is doing. I have downloaded the error log from the RAID controller but it too is just cryptic enough to be useless to me.

So, to recap... a previously operational RAID array in RAID 10 array on port 4-7
Western Digital Drives WD1002FBYS 1TB. Very poor disk performance and now getting hourly error message from the RAID controller in the system log. RAID controller shows the drives to be healthy.
Stumped.errorlog-0.txt

This message is the result of some 3rd party software trying to send a LOG SENSE command that the array does not support. It does not reveal the full command but it is typical of 3rd PARTY S.M.A.R.T. software. That program is trying to ask the RAID array itself not individual drives of the problem.

This command is NOT an error. This is what is supposed to happen when a SCSI emulated device gets a command it does not support. It is returning the proper data to whatever program sent the request to get log page info.

Are you running some S.M.A.R.T. software not written by 3ware? IF so uninstall it. The software won't work anyway, and it is causing this thing to happen in the first place.

Ahha, brilliant and so simple. Diskkeeper has been trying to check the drive health.
So that mysyery is solved, but it doesn't resolve my performance issues.
It too Diskkeeper a week of plugging away to finally defrag the drive and that has helped a great deal, but the read writes are still too slow.

Once again, checkdisk came up clean (although it took an hour to run)
and scf /scannow only made a few adjustments.
I know I am virus free, so what's left? I've never known a raid card to die a lingering death and with all of the drive useage I'm sure that if a drive was going to fail it would have done so... what's left?

Not the solution you were looking for?

IT issues often require a personalized solution. With Ask the Experts™, submit your questions to our certified professionals and receive unlimited, customized solutions that work for you.