So, ignoring the per character values, and knowing that this is a fileserver serving (primarily) music and video to the network, I think the most important benchmarks are the block based sequential reads/writes and rewrites. Which gives:

Having said that, the CPU percentage has increased, but that's OK if I can get better throughput. In addition, the latencies (although not the max per sec values) of the file creation are much lower.

I'm just restoring the data from the single disk and whilst I'm not able to read the data at 100% speed (I think rsync is causing a bottleneck there) the write speeds as stated by gstat seem to be much better (before I was only ever getting 40-50Mbytes/sec per disk writing with the interface maxed out, now I'm getting 100+Mbytes/sec and the interface isn't maxed out.

A nice and complete post containing clean presentation of a problem, analysis and solution. I would encourage you to drop a small memo what has been done, since many people will be concerned by this or related. Keep up the good work.

I'll report back later when the 2.5TBytes of data have been restored and I've run some more benchmarking on the re-setup disk. I also have a mirror of 2x1TByte Samsung drives that I might do the same to and see if the performance increases.

@arad85
First of all, congratulations! It feels good conquering technology
One quick about gnop though, it`s only necessary on the first drive in every vdev. So in your case with your raidz, you would only need the disk1.nop.

But.

arad85 said:

Running:

Code:

# zdb storage | grep ashift
ashift: 12

which means I'm 4096 byte aligned (2^12 = 4096).

Click to expand...

ashift has never been any kind of alignment! The ashift-value determines the smallest IO that ZFS will send; in this case 2^12 = 4096. And it would have kept sending 4k IO`s regardless of what alignment you may have had.

And for people reading this and wondering why it defaults back to using the device-names instead of the labels in zpool after re-import can try my approach instead, which doesn`t need export/importing and keeps the labels showing:4x2TB disk partition help

The procedure begins at post #12 and is for a bootable striped mirror pool, which you may change to a different pool layout to better suit your needs. Omit the first partition and bootcoding, plus zpool set bootfs if you don`t want to boot from it.

PS. The dd command from /dev/random showed no improvement in performance but more realistic benchmarks (bonnie++) seemed to show significant improvements.

Click to expand...

If I remember correctly, dd if=/dev/random is limited to less than 100MB/s.

Sebulon said:

And for people reading this and wondering why it defaults back to using the device-names instead of the labels in zpool after re-import can try my approach instead, which doesn`t need export/importing and keeps the labels showing:4x2TB disk partition help

The procedure begins at post #12 and is for a bootable striped mirror pool, which you may change to a different pool layout to better suit your needs. Omit the first partition and bootcoding, plus zpool set bootfs if you don`t want to boot from it.

ashift has never been any kind of alignment! The ashift-value determines the smallest IO that ZFS will send; in this case 2^12 = 4096. And it would have kept sending 4k IO`s regardless of what alignment you may have had.

Click to expand...

Of course... that makes sense.

Sebulon said:

And for people reading this and wondering why it defaults back to using the device-names instead of the labels in zpool after re-import can try my approach instead, which doesn`t need export/importing and keeps the labels showing:4x2TB disk partition help

Click to expand...

Ahh rats... I wanted to keep the labels, but you are right, you lose the labels. I now have adaXp1 as my drives.

Is there any way to get the system to see them as labels without having to rebuild the array (nearly finished transferring back the files, but if I must, I will...)? I'm guessing this may be important if I ever moved the disk array to a different controller (which I did a couple of months ago) as it doesn't then matter how they are physically connected up.

So, 3x reads, 4x writes across the 3+1 RAIDZ array from two controllers hosting all 7 disks and I get a sustained 275Mbytes/sec write rate. Not bad IMHO - and the disks aren't maxed out according to gstat