We created a monitoring report for IOPs on performance counters using Disk reads/sec and Disk writes/sec on four servers (physical boxes, no virtualization) that have 4x 15k 146GB SAS drives in RAID10 per server, set to check and record data every 1 second, and logged for 24 hours before stopping reports.

The average disk reads and writes per second were generally low. I.e. for one particular server it was like average 33 writes/sec, but when monitoring in real-time it would often spike up to several hundreds and also sometimes into the thousands.

Could someone explain to me why these numbers are significantly higher than theoretical calculations assuming each drive can do 180 IOPs?

If you're benchmarking using realistic activity patterns, and the application performance is acceptable and within the specs of the hardware, then you're in good shape. However, understanding your read/write patterns and the capabilities of your storage system is also important.

1). You have the write cache, which is just the physical disk cache on each drive. Maybe 8-32 megabytes. This is probably disabled in your setup.

2). You also have the battery-backed write cache (BBWC) or flash-backed write cache (FBWC) on the actual controller. This is either 512MB or 1GB, protected by a non-volatile cache mechanism. This appears to be enabled.

3). The cache ratio you described in your question is the percentage of the above dedicated to reads and writes. It's denoted by the "Array Accelerator" terminology.

By having the array accelerator enabled, you're going to have low-latency writes committed to cache before going to disk. Basically, your application can say, "yeah, I wrote that" because the storage system says, "it's written" and can coalesce writes and commit them to rotating disk in sequential batches.

You have a 384MB or 768MB write buffer, based on your current settings, so that accounts for the high IOPS figures during your test. You have a small amount of read cache available as well. If your working set of data is small enough, you could be working entirely in cache, which is far faster than disk.

Here's the output of a Smart Array P410 configuration on a ProLiant DL380 G7. As you can see, there's a lot involved in the basic setup, and there are a few optimizations. I think you may have only disabled one small item, leaving the rest in place.