At work I setup a pair of file servers and I would like to share the results because the topic of file servers has come up a few times in the past on this forum. Prior to the project there were a lot of prosumer NAS devices from LaCie and Seagate and a few hardware RAID 5/6 arrays on Windows Server. As needs grew more devices were purchased and thrown into the mix (another NAS here, another NAS there). This was a problem because stuff was hard to find, most stuff wasn't backed up nor would it be easy to backup, and most of the devices had fairly bad performance like the LaCie and Seagate NAS products.

So the goal was the opposite of that - high performance, scalable, backed up, and easy to manage storage. I picked ZFS on Linux because ZFS provides all of those things (and a lot more) and I'm familiar with Linux. Had I not been familiar with Linux I would've opted for ZFS through NAS4Free or FreeNAS which provide a web interface for management.

For the hardware I picked the SuperStorage Server 6047R-E1R36L from Super Micro since it's designed for software defined storage like ZFS. There are two, one for production and one for backup. It has 36 drive bays that are connected to a SAS controller and backplanes with switches but there's no RAID functionality (don't need it with ZFS). They have a few other useful features like redundant power supplies and hotswapable fans. From there you add processors, memory, disks, and network adapters.

No frills desktop drives were chosen since there's no hardware RAID controller and thus no need for error recovery control like TLER/CCTL/ERC which you find on "RAID" or "enterprise" edition drives so they don't drop out of arrays when trying to read bad sectors. At the time that meant a 4TB drive for $165 each, or a little less than half the cost of equivalent drives with error recovery control. When disks account for the largest single line item in a storage system that's a big deal.

Some other hardware details, went with Intel X540-T2 10GbE adapters since they work with Linux out of the box and have dual ports (one for the switch and one direct for backups). There are four SATA connected SSD, one for the OS (Ubuntu 12.04 LTS) and three for caching which ZFS calls ZIL (sync write caching) and L2ARC (read caching). There's 128GB of ECC memory which ZFS uses for the ARC (read caching). There's also a Netgear M7100-24X which is a 24 port 10GbE switch and threw in the optional redundant power supply. Clients use the Intel X540-T1 10GbE adapter which is a single port adapter and is widely supported in terms of operating systems (support on Mac through Small Tree).

In the end what does that get you? It gets you about 90TB of storage that can lose up to 12 disks, can easily saturate a 10GbE pipe, backs up automatically to an identical setup every hour, and has automated rolling snapshots so you can revert to older versions or recover accidentally deleted files. Once a week it scrubs itself for errors and automatically corrects any that it finds. External JBOD arrays can expand the system to 144 drives (about half a petabyte with 4TB disks) and it would continue to be in the same namespace and could even happen without rebooting the system.

The cost was around $35,000 for the pair of servers, a dozen client adapters, 10GbE switch, cabling, and eight spare disks. A few vendors bid on it and they ranged from $120,000 to $150,000 for similar setups though with less disk space (24 drive bays instead of 36). The setup has been running for more than a month now and has about 10TB of data on it. Clients typically see 300MB/s (2.4Gb/s) transfers which is limited by the disks on the clients.

Hopefully this is helpful information for somebody. For what it's worth ZFS scales down too, like an 8 bay or even smaller server. For scaling up I'd look at Lustre or GlusterFS on ZFS and move to a distributed cluster. If you have any questions or want more specific information let me know.

Hahaha, yes, it is indeed a Fry's Electronics bag. I had to get a few SATA cables and power adapters for the SSD which I didn't order with the rest of the parts (oops!). They are surprisingly quiet for what they are. You can hear the fans go full tilt when it boots but after that they calm down quite a lot. Either way it doesn't matter though because I'm far away from the racks!

Follow Us On:

The CGSociety

The CGSociety is the most respected and accessible global organization for creative digital artists. The CGS supports artists at every level by offering a range of services to connect, inform, educate and promote digital artists worldwide. More about us on TheArtSociety.com