I'm personally partial to FreeNas, mainly due to there being a commercial company behind it, that pays a larger amount of developers, the larger community (altho they have a large amount of anti-social members), and seem to have a healthier commit rate.

Whats the reasoning behind all SSD ? is it just reliability ? or does Speed factor into this as well ??

Click to expand...

We've used both FreeNas and Nas4Free. Both have been solid. I also helped a buddy deploy an IXSystems solution in a DC and those guys were great to work with.

SSD's are purely a reliability play. We're based on California, but our data centers are spread throughout the country. While we can always use the DC's remote hands service, we prefer to deploy systems that will be as reliable as possible. And since we don't need a lot of storage, SSD's aren't cost prohibitive.

Proxmox Subscriber

Before settling on FreeNas/Nas4Free you should consider Omnios + Napp-it too. You get native in-kernel ZFS and an iSCSI implementation which is more stable, scaleable and outperforms both istgt and cld. The integration with Proxmox is also better than the two others.

Proxmox Subscriber

Forgot to mention: If you use thin provisioned Zvols you should choose scsi disks and virtio-scsi controller because comstar (the iscsi daemon in Omnios) supports the scsi unmap command. This means the trim command is honored by comstar and therefore trimmed blocks will be released to the pool from the Zvol.

That seems to be the common consensus. Obviously we'll build the NAS to be as reliable as possible, but it seems like the component that would be the single point of failure to the entire cluster should be redundant.

remember murphy's law so always have the data mirrored to an other datacenter

how you will do firmware upgrade or something else on storage system where you need to reboot the controller? I don't think the solution is to stop all vms on all hosts. Simply switch to other datacenter and do maintaince on the local storage controller.

Proxmox Subscriber

That seems to be the common consensus. Obviously we'll build the NAS to be as reliable as possible, but it seems like the component that would be the single point of failure to the entire cluster should be redundant.

The reason i keep asking is as follows:
When ever you use a SAN, Ceph or Gluster you want to go with a separate Storage network, or a single network, that is properly sized and properly managed by QOS.

So lets say you have a 1G pipe and 2 Sans, your left with 0.5G or 62.5 MB/s of Bandwith outgoing from your Proxmox-Node, when you write from it. That is shared on 50 CT's. Thats why i asked earlier if you have any metrics to share on your current usage of your storage-subsystem.

It is also important the other way around. Lets say you have 2 Gluster Nodes, each with a single dedicated 1G pipe, And you have 3 Proxmox-Nodes attached to them. When only one Proxmox-Node is reading a large amount of files, you statistically end up with 2G worth of bandwith (or 250 MB/s for 50 CT's), but if all 3 proxmox-Servers are using the Gluster-Storage, you are looking at 2G/3=0.66G or 83 MB/s for 50 CT's. which is btw 1.6 MB/s per CT.

Not sure that will work, but thats why knowing your current metrics is important.

I'd self-build a node with Gluster before i'd go an buy a ready-Made san (and maybe set up Gluster ontop of it), or go with something like netapp.

A lot cheaper. The reason is not just base cost, but also running cost.
This is because you can leave out all the redundancy features, and spec it to exactly your needs all you need is case+mainboard+cpu+ram+psu+Disks/Flash+nic(s). You size em exactly as you need em for your use-case
Then just setup your favourite Linux + ZFS + Gluster and your done.
Need more redundancy ? just add another Gluster-node.

Q: Have you checked how much of your 8-12 TB data is Cold-Data and how much is Hot data ? Might make a big difference in terms of Cache-Sizing or the need of Raid-Levels on the NAS (Raiz2 / Raid10)

Am i reading this correctly ? You are doing <=600 Writes /s ? and no reads ?
That would mean your 3-Node Cluster does less then 2k Writes/Reads, right ?

The following is assuming you are doing the typical 5% ultra-Hot, 5% hot and 90% Cold-Data.

If this is true, i'd do a self build ZFS based NAS or Gluster setup with enough HDD's in RaidZ2 (min. 5x4 TB for your 12TB Goal) and dedicated SSDs for L2Arc and ZIL. I'd add at least 2 Spare HDD's and a minimum of 2 SSD for ZIL and L2Arc (256+ GB - no need to go crazy here) and make it all consumer-grade. This should be MORE then enough performance for what your iostat indicates and allow for growth in terms of additional IO.

Not having 24/7 easy access to the server and having to rely on DC support staff, might make me a bit weary tho, which might convince me to shell out the $$$ for enterprise-grade SSD', but even then i'd rather go with add additional spare SSD's and maybe a couple more spare HDD's. Consider that ZIL and L2Arc are adaptive, meaning that if they are not present, your setup will still work, but will only be able to rely on your RAM and unable to offload "overflow" to the ZIL / L2Arc.

In short you want at least:
2x OS-Drive
7x 4TB Disk (5x4TB for 12 GB goal with RaidZ2 + 2 spares)
2x 512GB SSD (so you can have one fail for Zil and l2arc)
Appropriate Sized RAM
should all fit into a 12-bay 2HE unit.

should still be cheaper then going all enterprise gear or even all enterprise Flash based.

Personal Note: we have all our servers in local datacenters (3), operated by our own staff (24/7/365), so we can switch failed drives rather quickly our selfs. We also rely heavily on redundancy in terms of additional servers and think of "pod-redundancy" instead of single -component redundancy. Thereby accepting multiple servers down at the same time due to failed Disks or PSU or network cards. Thats why we always go Consumer-Grade rather then enterprise-gear, dual PSU's, etc. A lot cheaper at scale. So keep that in mind when reading my suggestions

Quick Navigation

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick support. 13,000+ satisfied customers have Proxmox subscriptions. Get your own in 60 seconds.