What if you don’t need virtualization at all and your filesystem needs are as simple as “Mostly holds my data most of the time” ?

I realize that sounds flippant, but for those of us who work mostly with short lived cloud instances, it’s not clear to me whether these are significant enough advantages to warrant the increased difficulty one might encounter in building infrastructure using these operating systems.

Illumos and FreeBSD are amazing, it’s just not clear to me that they represent the clear and present broad based win the the article author is touting.

(I know there are a ton of BSD and Solaris folks here so I’m prepared to be tarred and feathered for my ignorance :)

At any rate Ted T’so (the ext[34] maintainer) has said as far back as 2009 that ext4 was meant to be transitional technology:

Despite the fact that Ext4 adds a number of compelling features to the filesystem, T’so doesn’t see it as a major step forward. He dismisses it as a rehash of outdated “1970s technology” and describes it as a conservative short-term solution. He believes that the way forward is Oracle’s open source Btrfs filesystem, which is designed to deliver significant improvements in scalability, reliability, and ease of management.

Btrfs has played a role in increasing efficiency and resource utilization in Facebook’s data centers in a number of different applications. Recently, Btrfs helped eliminate priority inversions caused by the journaling behavior of the previous filesystem, when used for I/O control with cgroup2 (described below). Btrfs is the only filesystem implementation that currently works with resource isolation, and it’s now deployed on millions of servers, driving significant efficiency gains.

Ext4, like most evolutions of existing filesystems, is strongly constrained by what the structure of on-disk data and the existing code allows it to do. Generally there is no space for on-disk checksums, especially for data; sometimes you can smuggle some metadata checksums into unused fields in things like inodes. Filesystems designed from the ground up for checksums build space for checksums into their on-disk data structures and also design their code’s data processing pipelines so there are natural central places to calculate and check checksums. The existing structure of the code matters too because when you’re evolving a filesystem, the last thing you want to do is to totally rewrite and restructure that existing battle-tested code with decade(s) of experience embedded into it; if you’re going to do that, you might as well start from scratch with an entirely new filesystem.

In short: that ext4 doesn’t have checksums isn’t surprising; it’s a natural result of ext4 being a backwards compatible evolution of ext3, which was an evolution of ext2, and so on.

It continues to baffle me how “mainstream” filesystems like ext4 forgo checksumming of the data they contain. You’d think that combatting bitrot would be a priority for a filesystem.

Ext4 doesn’t aim to be that type of filesystem, for desktop use on the average user, this is fairly okay since actual bitrot in data the user cares about is rare (most bitrot occurs either in system files or empty space or in media files where the single corrupt frame barely matters).

If you want to check out a more modern alternative, there is bcachefs. I’ve been using it on my laptop for a while (until I stopped but now I’m back on it) and it’s been basically rock solid. The developer is also working on erasure coding and replication in a more solid way than btrfs currently has.

Newer features like encryption and raw sends (likely crucial to some end-to-end encryption features Datto is planning on offering; Tom Caputi, one of their engineers, authored that feature) might take a while to land in FreeBSD and IllumOS.

See http://open-zfs.org/wiki/Features for more details - it seems to be a bit out of date, especially regarding things like encryption which made their way to 0.8.0-pre in Linux.

So, the result is that a lot of newer pool feature flags are Linux-only for now, and pools created with -pre releases of ZFS can’t be imported on other platforms. Not a problem if you’re only using Linux, but something to keep in mind.

We’re working on importing the encryption changes, but we’re not going to finish that process until we’re sure they’re ready. There have been some panics and other issues that we’ve been chasing down. It’s the file system, so it’s extremely important to get this stuff right, even when it takes a while.

I’m not sure that I’m scared. I think the Lua interpreter we have is almost certainly a high quality piece of code that has seen a lot of testing and (I believe?) fuzzing in the upstream codebase. I definitely had (and, really, still have) reservations about it – you can read about some of them in comments I made on the pull request.

That said, because the implementation is still (as far as I know) constrained to use by the super-user – and even then, only in the global zone – I don’t think it’s been the cause of much if any havoc on general purpose or multi-tenant systems. I do appreciate that if you’re building a sealed appliance that happens to sit on top of illumos, it probably does help with certain internal ZFS administration functions that you might want to build.

Not a single mention of DTrace? Why? For me personally that would be the biggest selling point, yeah it comes even before ZFS for me. It totally changed the way how I perceived operating systems and monitoring.

With pkgng, the package management tool used in FreeBSD has almost 27.000 compiled packages for you to use. Almost all software found on any of the important GNU/Linux distros can be found here. Java, Python, C, C++, Clang, GCC, Javascript frameworks, Ruby, PHP, MySQL and the major forks, etc. All this opensource software, and much more, is available at your fingertips.

But (last time I checked) unfortunately not Racket, which for me is a no-go for putting it on my server.

Tried using it on the desktop as well, but LibreOffice crashed constantly and Gnumeric gave strange rounding errors. Probably not the fault of the FreeBSD developers, but I think it’s best to be realistic about the trade-offs of using FreeBSD, which is not something that can be said about the documentation or the linked article:

PS: I haven’t mentioned both softwares, FreeBSD and SmartOS do have a Linux translation layer. This means you can run Linux binaries on them and the program won’t cough at all.

Sure. How much work do you think needs to be done to get the benefits of Docker’s layer-based approach to containers? If your containers are based on each other, you get significant space savings that way.

ZFS does not dedup by default, and deduping requires a lot of ram to the point that I’d not turn it on for performance reasons. I tried a 20TiB pool with/without, the speed was about 300k/s versus something closer to the underlying ssd’s performance. It was that bad, even after trying to tune the piss out of it.

One option is to use a jail-management system built on top of the raw OS functionality. They tend to take an opinionated approach to how management/launching/etc. should work, and enforce a more fleshed-out container model. As a result they’re more ergonomic if what you want to do fits with their opinions. CBSD is probably the most full-featured one, and is actively maintained, but there are a bunch of others too. Some of them (like CBSD) do additional things like providing a unified interface for launching a container as either a jail or a bhyve VM.