Choosing Opensolaris & ZFS

After beeing a long term linux user on my primary server at home. I decided to switch. I begun running out of space on my existing machine, which was running gentoo linux with a RAID-5 Linux md software raid @ 750GB netto capacity and served me well for the last 3 1/2 years.
Well I could have just put in more or bigger hard drives and not make such a big fuzz. But since i saw a presentation of Sun at the Linuxtag 2006, i was intrigued by the capabilities and design of ZFS.
And with all the attention ZFS was getting recently, i began monitoring it’s progress in evolving more thoroughly by reading the zfs-discuss mailing list, and studying all websites, blog-posts and scientific papers I could get my hands on …
The reasons for choosing ZFS :
ZFS has an impressive feature list, and these features are being discussed in length everywhere. So I just want to mention the most important to me :

Continuuous Data Protection with checksumming.

Various “Raid” Level implementations, that don’t have a write hole.

Snapshots and Clones at block-level

Easy built-in sharing via NFS, CIFS (Samba) and iSCSI.

These features make it almost ideal for my use and in my opinion the ideal case for a non-distributed server filesystem, and far superior to ext2/3/4, reiserfs, xfs, jfs, ntfs or UFS. The only one that could compete is btrfs which also has a great feature list, but unfortunatly is not that far in it’s stage of development altough it was recently commited to the mainline linux kernel – the on disk format is not fixed yet (Sep. 2009).
This does not mean that ZFS is perfect, it’s probably far from yet. It has several quirks and shortcomings, among them :

You can not easily expand to an existing raidz zpool by adding a single harddrive
This means if you have an existing zpool, for example a 3 x 500GB raidz1, you can not add another 500GB disk to the zpool to expand it’s capacity. This feature is very often requested from home users, but not on a very high priority schedule for the developers, because big corporations tend to add complete raidzX vdevs to their pools.

Recovering your data can be pretty ugly
Of course you should always have a recent backup of your data. But once your zpool gets corrupted for whatever reason, and you have not backuped all the data, e.g. the data from the last two days, it can be a very complicated task messing around with zdb and other tools to get your data partly back, if it’s possible at all (corruption of entire pools actually can happen in spite of the transactional copy on write approach – see the zfs mailing lists – but I would still consider to be my data more safe with zfs than anything else).

The reasons that drove me clearly towards Opensolaris :
After beeing a long term linux user, I first took a look at what was the status of ZFS on linux. Well due to licensing issues between the CDDL and the GPL there will be appearently no kernel ZFS solution for Linux until the zfs licensing changes (which could be never, but considering Oracle buying Sun, there might be a glimpse of hope for the Tux). So you are left with using ZFS over FUSE (Filesystem In Userspace), which works ok if you want to play around with zfs but has nowhere near the performance of a native implementation. So using linux was not an option.
I had also previously touched various bsd flavors but never became too familiar with them. Well it might have been time to and after investigating further it turned out that FreeBSD has a real good ZFS implementation, with almost all the features working and good performance.
But for the fact that my primary server has mainly two functions (1) Storing and serving all my data and (2) acting as a Host for the Virtualization of a few machines which have assigned individual tasks and all run several services – FreeBSD wasn’t much of a choice either.
I had been running the free vmware server for that for years, and it served me quite well on my linux server, but running vmware on FreeBSD is unsupported and has at the time beeing some constrains and known bugs.
As I had always considered switching to Virtualbox, I evaluated that one but even virtualbox support is kind of inchoate and not officially supported on FreeBSD.
That left me with Solaris / SXCE or Opensolaris. After digging through the differences it quickly became obvious that OpenSolaris was my choice. It’s free, open-source, has a good community and probably the best ZFS support of all operating systems, as the newest zfs features and bugfixes always get there in svn first. Also the Virtualbox support on Solaris is of course (for Sun now beeing the owner of VB) great.
So I checked on the OpenSolaris HCL (Hardware Compatibility List) and the Mailing lists for the compatibility of my present server hardware.

AMD 780g Motherboard (Asus M3N),
onboard NIC is not directly supported, but through a third party driver. If this solution would not work as expected, I would just put in a pci-express intel NIC.
Onboard AHCI should work ok with recent opensolaris builds.

Athlon 64 X2 4850e,
one small problem here – cool and quiet power management is not supported under OpenSolaris, so now frequency scaling, but no too big issue since I plan moving on to an Athlon II or Phenom II, for which cool n quiet is supported (for all AMD K10 and above)

Icydock MB-454-SPF, 4-port sata Bay,
well this is just the enclosure, no driver needed – no problems

Dawicontrol DC-4300 (Silicon Image sil3124),
good OpenSolaris support – yes NCQ should work