I use ZFS on FreeNAS, and I was thinking about switching to ZFS for pfSense as I'd really love to be able to roll back an update if I don't like it.

How much of a drag a ZFS boot drive would be on this system? The FreeNAS project says don't even think about FreeNAS (ZFS Only-No UFS anymore) below 8GB.

Given how full the SSD is, I either need to drastically cut the disk usage on ntopng, (it's sucking up about 60GB), stop using it all together, or buy a 256GB SSD. I'm not williing to spend more money on this box, so if it won't pass muster, I'll have to wait for my next box before going ZFS.

I suspect that is meant for those using deduplication, which is RAM-intensive.

It's needed for ARC. Apparently guys were losing their pools and having other odd things going on with less than 8GB so they upped the minimum requirement to 8. If people stick to that the weird issues seem to go away.

I suspect that is meant for those using deduplication, which is RAM-intensive.

It's needed for ARC. Apparently guys were losing their pools and having other odd things going on with less than 8GB so they upped the minimum requirement to 8. If people stick to that the weird issues seem to go away.

I'm wondering have any similar problems been experienced with pfSense?

A pool with 8 x 8TB (or larger) pool with some heavy file i/o is certainly going to need a lot more ARC than a pfSense box.

Have any problems been reported that are attributable to ZFS?

I would think that ZFS might kick the crap out of a USB drive a lot faster than UFS - not my problem, because I'm using an SSD, but something to be mindful of.

The quoted values seem very wrong to me but there's some information missing. You don't need any certain amount of ARC cache for a pool of certain size for ZFS to operate properly, it will still work fine if you allocate too little ARC cache but you won't be getting the maximum read performance that you could get out of the pool.

On top of that, large amounts of ARC cache are for heavy fileserver use, on a firewall/router/proxy use I don't see where the system would be doing large amount of read operations simultaniously.

A feature of ZFS that doesn't get mentioned often enough is the use of boot environments (BEs, beadm is the tool used to administer them). Best way to upgrade a system I've ever seen. If it fails in some manner, easy to fall back to a known good one. I'd bet one could even automate a fall back if something in the new environment fails. It's very easy to create mirrors with ZFS so you can get a lot of redundancy easily. Yes, you can create mirrors for UFS using geom, but not too many folks do that. Of course your system needs the ability to have multiple devices.

Amount of RAM is very dependent on use case as others point out. If you look at what a pfSense or any firewall is doing, they boot, read configuration, start up processes. After that the bulk of what happens is writing to logs and "in memory stuff" (travsering state tables and such). There are standard FreeBSD packages that let you look a the zfs usage stats, so you could keep an eye on things.

ZFS by default will use system RAM so there may be some competition on that resource, but ZFS lets you tune and set limits. ZFS also groups transactions (mostly reads) that get batched out to the device, so you may have periodic spikes in CPU usage because of this.

(BEs, beadm is the tool used to administer them). Best way to upgrade a system I've ever seen

This would be the main factor for sure… So your saying I could take say a snapshot/mirror before I do an upgrade and if something goes wrong just boot the previous boot environment?

So I have a sg4860.. Its got a 32GB eMMC and 8GB of ram.. why would it not have shipped with zfs if would add the ability for boot environments.. This would be fantastic option when upgrading or if wanting to play with the dev snapshots. Most say cisco switches do this where when you update the firmware you upgrade the other image and pick which image you want to boot, etc. Riverbeds do it the same way, etc.

It may depend on what version of FreeBSD you're basing on. The TrueOS project (ixSystems guys, PC-BSD followon) are basing on 12-CURRENT and a ZFS install and they have been doing binary upgrades into a new BE since day 1. 11-Release should support them just fine.

The process is basically create snapshot and clone of the currently running environment and the upgrade is done into that clone. Then you readjust flags and reboot into that upgraded BE. If anything fails, the fallback is either:
reboot, stop in bootloader, select previous BE and continue booting
or
if the system is up enough, use the beadm command to activate the previous BE and reboot.

There are some rules you need to follow when creating any datasets (use of the canmount property to make sure things wind up where they need to be), but it should be doable. This is the part that gets a lot of people, things needed for booting have to fall under whatever root dataset you create. Just remember to limit the number of BE's so you don't run out of disk space.

ZFS by default will use system RAM so there may be some competition on that resource, but ZFS lets you tune and set limits. ZFS also groups transactions (mostly reads) that get batched out to the device, so you may have periodic spikes in CPU usage because of this.

This needs more qualification. ZFS isn't per se much different than UFS for example in that it does use some system memory for its bookkeeping data structures. What is different is that ZFS allocates the ARC cache from the so called wired memory, the non-swappable kernel memory.

The grouping of transactions applies only to writes really because if there are only reads there is no need for transactions to ensure data integrity. It is of course possible that in some cases a write transaction blocks a read to the same data. Write performance, especially in applications like NFS, is one of ZFS's achilles' heels because it goes to such lengths to try to guarantee that synchronous writes are truly atomic.

So getting into the weeds, yes, ARC is nominally kernel memory, accounted for in the Wired statistics in top, but by design it will expand to a threshold of system memory. Wired and nonswappable does not mean unrecoverable.
There are boottime tunables to change this behavior (vfs.zfs.arc_max, vfs.zfs.arc_min)

So if you have 32GB of RAM in a system and your use case warrants it, ARC certainly will use I think up to all but 1 or 2 GB reserved for the kernel. But if other processes demand RAM, ARC will flush and use less. Plenty of documentation on this, but the best I've seen is the Michael W Lucas books on ZFS.