What do we want? Data integrity! When do we want it? Now! —

ZFS-loving Mac users demand support in OS X 10.9

You can sign an online petition, but it may be too late to sway Apple.

Some Mac users are demanding that Apple add modern file system support in the next major version of OS X. An online petition has been started to let Apple know that its aging HFS+ file system just won't cut it any more, and the company should include ZFS in OS X 10.9, expected later this year.

HFS+ is the current file system used by OS X (and iOS). It was originally developed as HFS, or "Hierarchical File System," for the original Mac OS in the early '80s. A team at Apple, led by engineer Don Brady, adapted HFS for 32-bit systems in the mid-1990s. Brady later adapted HFS+ to work with the UNIX environment that OS X was built on, and over time he and other Apple engineers added additional features, including the extensible metadata used by Mac OS X's Spotlight search, live partition resizing used for Boot Camp, and the Adaptive Hot File Clustering used to reduce seek times for oft-used system files.

Despite all the features Apple has managed to tack on to HFS+, though, its design certainly isn't modern. "The initial HFS+ was primarily about addressing the block count problem," Brady told Ars in 2011. "Since we believed it was only a stop-gap solution, we just went from 16 to 32 bits. Had we known that it would still be in use 15 years later with multi-terabyte drives, we probably would have done more design changes!"

ZFS, on the other hand, was designed from the ground up to address the ever-increasing needs for large amounts of storage, as well as the need to protect data as it is written to and read from disk. As a 128-bit native file system, ZFS can address up to a theoretical 256 quadrillion zettabytes. One zettabyte alone is equivalent to over a billion terabytes; 256 quadrillion billion terabytes is more storage space than could practically be used on Earth. It also includes several features designed to ensure the integrity of data on the disk, including checksumming every block of data so the system knows if a block goes "bad" and RAID-like features that allow disks to "heal" themselves if data corruption is detected.

Apple actually flirted with ZFS early on in its development. Brady was involved in a "skunkworks" project to port ZFS to OS X that started in 2005. Some of the code shipped in Leopard (10.5) and was expected to be a major feature of Snow Leopard (10.6). But due to licensing issues with Sun (and perhaps other reasons), Apple dropped all support for ZFS in Snow Leopard and cancelled the open source project that had served as official support for ZFS on OS X. (That project was forked and still exists as MacZFS.)

Brady later left Apple and started his own company to build a commercial version of ZFS for OS X. He got as far as releasing a command line "community" version called ZEVO before his company was acquired by enterprise software maker GreenBytes in June 2012. GreenBytes still offers the free community version of ZEVO while Brady continues to work on a GUI version that can integrate with OS X (though restrictions like sandboxing have proven difficult to work around).

Still, some users want official support for ZFS "or its equivalent" from Apple, and they want it soon. Mac user Thomas Monte started an online petition practically demanding that Apple add modern filesystem support to OS X 10.9.

(The petition also asks Apple to update the ancient OpenGL support in OS X, which still lags Windows significantly, from version 3.2 to the latest 4.3. Previous sources have indicated that improved OpenGL support is indeed coming.)

Unfortunately, the demand for ZFS support may fall on deaf ears. Aside from the fact that Apple hasn't shown any indication that it will support anything other than HFS+ for the time being, OS X 10.9 is already showing signs that it is being widely tested internally at Apple. It is also likely to get its first public showing in the next several weeks, and if Apple keeps to its projected one-year development cycle, it could be released this summer.

Still, it's been 30 years since Apple originally developed the basis for the file system currently used in OS X. Whether Apple adopts ZFS, Oracle's BtrFS, or is secretly rolling its own modern file system, OS X is long overdue for something new.

Full data checksum verfication, snapshotting, and atomic writes justify ZFS even for single drive computers.

A lot of the features that make a modern filesystem great and useful for single-disk / regular users consume disk space (deduplication excluded, I guess), but we are in a weird transitional period with SSDs, where most macs (laptops) don't really have disk space to spare. It might be another couple of generations before we are back the the affordable terabyte region where normal folks are not constantly running into the limits of their small SSD, at which point the fact that ZFS snapshots can be exposed as a fully local Time Machine view will be awesome.

172 Reader Comments

I don't use a Mac and have no opinion on ZFS per se. But the idea of cramming a new file system into an OS update, as a last minute decision, seems like it would be a huge mistake. Maybe ask for it as an OS X 11 feature, but not as an add-on for an update that's already hit testing.

Oh and stop making stuff like the Library folder hidden. People who shouldn't be messing with it don't even know it's there, it just makes it a pain in the arse when I need to go there.

Hold down option in the Go menu, or if you know what you're doing just change the visibility on the folder to make it show up again.

Seriously, it is that easy. I think Apple did the right thing in this case. Normal users should not be moving or changing anything in Library, but Library is named like something where you go to get a document you want to read. It is not.

Ever increasing drive sizes means data rot is becoming a legit problem even for users that only have a single 2 or 3TB drive.

I am still curious whether this is actually a real problem or purely theoretical (or at least blown way out of proportion); I'm certainly not disputing that hard drives can develop bad sectors, but given that the hard drive itself uses ECC for each block, I am not convinced that bit rot could occur in stationary data on disk without also triggering a read error (as opposed to just silently returning corrupt data).

My guess is it’s simply the fact that HFS+ seems to work. Heck, they even did Fusion Drive on top of it—that doesn’t sound like it’s having problems "keeping up".

Is there any technical reason why HFS+ couldn’t be, er, refactored, for lack of a better term?

Walt French wrote:

The bigger problem may be that Apple has more revenue coming in from iTunes than from Macs.

I don’t think that’s as big a problem as many believe. Tim Cook doesn’t strike me as someone who’d let a multi billion dollar business with hundreds of millions of profit wither and die just because it doesn’t compare well to a ridiculously successful business of selling iphones or ipads. Plus, I’m pretty sure that developments on either side, iOS and OS X, are mutually benefitial. The only problem I’d be afraid of is having too few competent engineers to provide solid software on all the fronts (OSes, iWork, iLife, A/V apps, etc.) in a timely manner.

Oh and stop making stuff like the Library folder hidden. People who shouldn't be messing with it don't even know it's there, it just makes it a pain in the arse when I need to go there.

Hold down option in the Go menu, or if you know what you're doing just change the visibility on the folder to make it show up again.

Seriously, it is that easy. I think Apple did the right thing in this case. Normal users should not be moving or changing anything in Library, but Library is named like something where you go to get a document you want to read. It is not.

Bad Superblock has it right. For power users, there are even Terminal commands that allow you to see ALL hidden files. I find it useful when doing local Web development (the .htaccess files don't disappear from view), but turn it off the rest of time.

My guess is it’s simply the fact that HFS+ seems to work. Heck, they even did Fusion Drive on top of it—that doesn’t sound like it’s having problems "keeping up".

Is there any technical reason why HFS+ couldn’t be, er, refactored, for lack of a better term?

Walt French wrote:

The bigger problem may be that Apple has more revenue coming in from iTunes than from Macs.

I don’t think that’s as big a problem as many believe. Tim Cook doesn’t strike me as someone who’d let a multi billion dollar business with hundreds of millions of profit wither and die just because it doesn’t compare well to a ridiculously successful business of selling iphones or ipads. Plus, I’m pretty sure that developments on either side, iOS and OS X, are mutually benefitial. The only problem I’d be afraid of is having too few competent engineers to provide solid software on all the fronts (OSes, iWork, iLife, A/V apps, etc.) in a timely manner.

And if it truly were a benefit to add ZFS, compared to all alternatives, I can't see why Apple wouldn't do it--even if iTunes centric. Huge data pools would be an incentive for users to acquire more media--hence helping the iTunes franchise.

And if it truly were a benefit to add ZFS, compared to all alternatives, I can't see why Apple wouldn't do it--even if iTunes centric. Huge data pools would be an incentive for users to acquire more media--hence helping the iTunes franchise.

Ever increasing drive sizes means data rot is becoming a legit problem even for users that only have a single 2 or 3TB drive.

I am still curious whether this is actually a real problem or purely theoretical (or at least blown way out of proportion);

It's a real problem, not theoretical at all. I've seen it personally in both business and personal datasets stretching back across the last 15-20 years, which makes sense since mathematically it becomes a near certainty with typical consumer drives once the amount of data passes a couple of terabytes. It's often easiest to detect in images: I've seen a number of old photos that were opened back up after many years and were corrupted, stuff I know absolutely 100% were fine in the past. Backups are hit or miss there because of course they'll happily backup corrupt data, they don't know any better (and for that matter it's possible for data on the backup to become corrupt, or to get corrupted during transfers, etc). I've yet to see it cause any truly major damage, but it's certainly been a real irritation for quite a while, and there have been a few cases where a photo did simply have to be written off.

There's plenty of higher level hacky workarounds of course, but frankly as digital data becomes ever more important it's simply not something we should even be having to think about in 2013. We've got gobs of processor power, memory, and storage available, and there just isn't any reason why the filesystem shouldn't be taking care of making it all extremely reliable. Higher level applications shouldn't need to worry about it, much less users. The other features of ZFS are very handy too, but that's enough to justify itself IMO.

Ever increasing drive sizes means data rot is becoming a legit problem even for users that only have a single 2 or 3TB drive.

I am still curious whether this is actually a real problem or purely theoretical (or at least blown way out of proportion);

It's a real problem, not theoretical at all. I've seen it personally in both business and personal datasets stretching back across the last 15-20 years, which makes sense since mathematically it becomes a near certainty with typical consumer drives once the amount of data passes a couple of terabytes. It's often easiest to detect in images: I've seen a number of old photos that were opened back up after many years and were corrupted, stuff I know absolutely 100% were fine in the past. Backups are hit or miss there because of course they'll happily backup corrupt data, they don't know any better (and for that matter it's possible for data on the backup to become corrupt, or to get corrupted during transfers, etc). I've yet to see it cause any truly major damage, but it's certainly been a real irritation for quite a while, and there have been a few cases where a photo did simply have to be written off.

There's plenty of higher level hacky workarounds of course, but frankly as digital data becomes ever more important it's simply not something we should even be having to think about in 2013. We've got gobs of processor power, memory, and storage available, and there just isn't any reason why the filesystem shouldn't be taking care of making it all extremely reliable. Higher level applications shouldn't need to worry about it, much less users. The other features of ZFS are very handy too, but that's enough to justify itself IMO.

This.

More or less, the issues are becoming the same as for film and negatives. Being in the creative field, it's becoming more vital. Most content today originates as digital, and stays that way.

For some important stuff, I burn to DVD for archival purposes. It gets to be a hassle, at times, though...

Is there any technical reason why HFS+ couldn’t be, er, refactored, for lack of a better term?

They already did that once from HFS to HFS+. Just as Ext2 was extended to Ext3 and then Ext4. But there is are practical limits on how much you can extend/improve an FS before it is cheaper easier and safer to build something new. The most modern filesystems, like ZFS, BTRFS, and Hammer are extraordinarily different from previous generation FSs like Ext and NTFS.

And if it truly were a benefit to add ZFS, compared to all alternatives, I can't see why Apple wouldn't do it--even if iTunes centric. Huge data pools would be an incentive for users to acquire more media--hence helping the iTunes franchise.

Apple uses iCloud and iTunes Match for that.

For most, that may work...

For those of us with FLAC and other types of lossless content, though--it isn't feasible...

Ever increasing drive sizes means data rot is becoming a legit problem even for users that only have a single 2 or 3TB drive.

I am still curious whether this is actually a real problem or purely theoretical (or at least blown way out of proportion);

It's a real problem, not theoretical at all. I've seen it personally in both business and personal datasets stretching back across the last 15-20 years, which makes sense since mathematically it becomes a near certainty with typical consumer drives once the amount of data passes a couple of terabytes. It's often easiest to detect in images: I've seen a number of old photos that were opened back up after many years and were corrupted, stuff I know absolutely 100% were fine in the past. Backups are hit or miss there because of course they'll happily backup corrupt data, they don't know any better (and for that matter it's possible for data on the backup to become corrupt, or to get corrupted during transfers, etc). I've yet to see it cause any truly major damage, but it's certainly been a real irritation for quite a while, and there have been a few cases where a photo did simply have to be written off.

There's plenty of higher level hacky workarounds of course, but frankly as digital data becomes ever more important it's simply not something we should even be having to think about in 2013. We've got gobs of processor power, memory, and storage available, and there just isn't any reason why the filesystem shouldn't be taking care of making it all extremely reliable. Higher level applications shouldn't need to worry about it, much less users. The other features of ZFS are very handy too, but that's enough to justify itself IMO.

Yes. I couldn't agree more. I don't know if the solution is ZFS or something else, but IMHO "THE ABSOLUTELY NECESSARY FEATURE" is end-to-end integrity. So much of our lives is in digital form it is completely unacceptable to have everything teetering on the edge of a cliff in a rickety shack. Worse, is you never really know if the shack has gone off the cliff until much later--silent corruption.

Hmm... I thought that a lot of why ZFS hasn't been adopted (or won't be) had to do with how ZFS requires the 'flush cache' command to actually work, and how a whole lot of consumer (and maybe enterprise, too) hardware just ignores it.

EDIT: Typos, typos, everywhere, and all the errors did blink. Typos, typos, everywhere, and not a spellchecker in use. (Horrible, so sue me.)

More or less, the issues are becoming the same as for film and negatives. Being in the creative field, it's becoming more vital. Most content today originates as digital, and stays that way.

It's worth emphasizing that it is all a probability game, most people have a significant amount of data where they'd never notice a problem, or any problem that did arise wouldn't be a big deal (an application gets messed up? reinstall it). It'd be perfectly possible to go decades and never have a real issue despite having a certain amount of bitrot. But by the same token, it's perfectly possible to never ever backup anything and never lose a drive. Failure rates aren't high, accidents are rare. If one day you do roll a natural 1 though...

Given the inevitable comparisons to Time Machine, it's worth noting that is one area where a GUI really would be critical for widespread adoption and important in general, and that it's also an area where it would really help to have a mainline project with changes merged in. In October 2011 Delphix did a presentation call "ZFS: Life After Oracle"(PDF) with ideas on important improvements to the ZFS replication process, particularly for consumer settings. That sort of thing would need to be there.

ZFS versioning and replication are more powerful, flexible and reliable then what Time Machine offers, but it's not like it would just be zero-effort adoption either.

There is no way that any team could end up with secure (secure for your own data) new filesystem implementation in less than a year. And ZFS is "badass" file system designed for many complex use cases hence its complexity.

Apple would not only have to bring it to OSX but also assure that it can be drop in replacement for HSF+ (so no part of OSX stop working cause of missing pieces..).

Lots of work, and too little time.

Just perfect petition from well intentioned people who do not know what are practical ramifications of their wish...

Full data checksum verfication, snapshotting, and atomic writes justify ZFS even for single drive computers.

A lot of the features that make a modern filesystem great and useful for single-disk / regular users consume disk space (deduplication excluded, I guess), but we are in a weird transitional period with SSDs, where most macs (laptops) don't really have disk space to spare. It might be another couple of generations before we are back the the affordable terabyte region where normal folks are not constantly running into the limits of their small SSD, at which point the fact that ZFS snapshots can be exposed as a fully local Time Machine view will be awesome.

It's a real problem, not theoretical at all. I've seen it personally in both business and personal datasets stretching back across the last 15-20 years, which makes sense since mathematically it becomes a near certainty with typical consumer drives once the amount of data passes a couple of terabytes. It's often easiest to detect in images: I've seen a number of old photos that were opened back up after many years and were corrupted, stuff I know absolutely 100% were fine in the past.

The probability of uncorrectable bit errors becomes high, but I don't think that's quite as true for undetectable bit errors (especially for disk sizes from 15-20 years ago). As long as the error is detectable by the sector ECC, you don't have to rely on FS-level checksumming to be able to recover (assuming you have sufficient redundancy).

If you look at the way that the entire Mac line is going, it just doesn't really make sense to go through all the work and validation required in putting together a new file system for your OS. All the laptops/AIO's are going to solid-state storage of < 1TB, Apple doesn't offer a server line any more, and their Mac Pros are pretty much the only case where a large multi-TB/multi-drive setup would appear.

In the meantime, you'd have to separate the iOS from the OSX - which is exactly the opposite of the direction they're looking to go (a single, unified OS) because such a system would be overkill in any single-SSD system, let alone any iOS device.

Are there features which would be nice? Yup. Are there scenarios where such a setup would be very useful? Indeed, it would. Are they really -necessary-? No, since they can easily be filled by different raid solutions for those special use cases where redundancy is necessary.

I'm sure Apple has the resources to create their own new filesystem if they though it was necessary. They'd never rely on any third party filesystem that Apple would not be able to control. But so far, there isn't anything terribly wrong with HFS+.

I love my Mac but I feel like OS X is starting to stagnate. I bought it when Leopard came out and it felt like the Mac was really picking up the pace and would be a great platform with some interesting developments. It is a great platform and it's still my OS of choice (and will be for the foreseeable future), but unfortunately all that seems to have happened is Apple poking it with a stick until it awkwardly shuffles a bit closer to iOS with every release. Stuff like Notification Centre: good. Stuff like Launchpad: wholly unnecessary. Speaking of which, iOS has the same feel. Here's hoping Apple can continue to improve their platforms with something new and fresh.Oh and stop making stuff like the Library folder hidden. People who shouldn't be messing with it don't even know it's there, it just makes it a pain in the arse when I need to go there.

I second that. I'm a recent convert to Mac and OSX, and I like it fine, I guess. But it's not any better than Windows 7, and - SHOCK - it's less fun to use than Windows 8. Say what you will about Microsoft, but at least they're pushing the envelope and making OS's which progressively boot faster and such. After updating from OSX Lion to Snow Lion, my shut down time went from five or six seconds to thirty or forty seconds. Boot ups have similarly slowed down, although they were never all that quick to begin with.

My iPad is becoming progressively less useful to me, too. My Windows Phone plays pretty much everything on the web, including Silverlight files necessary for classes I'm taking. My iPad is effectively worthless, unless you want to work in the hobbled app which was developed for it (and can't play the videos). I can get a quick snap shot of everything happening by looking at my phone's home screen. My iPad hasn't gotten much better in years.

I'm sorry to say it but I feel increasingly strongly that I jumped on the Apple bandwagon at the wrong time. I hope they do something to blow me away soon, because if not I'm actually looking forward to moving away from them at my next tech purchase (which won't be for quite a while). But my phone blows away my iPad, and was a tiny fraction of the cost.

Full data checksum verfication, snapshotting, and atomic writes justify ZFS even for single drive computers.

[…]These nice-to-have features don't justify swapping out a filesystem. If you want to change filesystems, you have to:[…] Ensure all of your OS features are reworked for the new filesystem (given how much is tied into it - Spotlight, Time Machine, etc)[…]

Time Machine: multi-linked directories <shudder>

That whole list is pretty darn relevant. There has to be a really big payback for the customer, because there's a potentially huge bag of hurt there for Apple and early adopters. With (probably) no new revenue for Apple to pay for the development effort, since this is all under the hood stuff.

Apple needs to do something about it's filesystem, but who's to say ZFS is the correct choice at this point. For one thing physical storage and the way data is used has changed pretty dramatically in the last few years. Perhaps Apple is splitting the monolithic "Filesystem" problem into, a two front battle, or into multiple fronts. Some problems might be dealt with at the filesystem level, and others at a higher level. Apples recent work with the Fusion drive might suggest this strategy.

A group of users can't demand anything of Apple or any other company in terms of their technology plans. Particular when your attempting to christen a particular technology from a far. It's like giving someone dancing instructions when your only contact, with them, have been through a walkie-talkie and a pair of binoculars. But I do think that the Mac community should voice it's concerns, and petitions are beautifully democratic. Perhaps it would be more useful to voice concerns about the problems an archaic tech causes, then to hand pick it's successor.

I could not disagree more. The fact that a corrupted file in HFS+ will overwrite a good backup on a Time Machine drive is horrible.

Time Machine doesn't overwrite the old versions, seeing as how that'd ruin the point of the Versions feature.

A corrupted file system will make your Time Machine backup unusable. It's happened to me. Nothing to do but wipe out 1+ years of Time Machine backups and start all over again. He's right HFS+ must go.

If you're depending on a single file system then you're just doing it wrong. The correct approach to this "bit rot problem" is to create more distinct copies of your backups than creating a single better backup.

That's why everyone proclaims "RAID is not a backup strategy". It isn't and neither is ZFS.

I'm still holding my breath for OpenGL 4.x and drivers that work with off the shelf video cards, after I pass out I'll be sure to put this on the list. I'm amazed that Apple is even on the Khronos board.

That was my first thought. The MySQL and OpenOffice debacles demonstrate that Oracle aren't exactly a team player, and depending on code that belongs to them might not be wise.

I agree that MacOS really could use something more modern (though to be fair HFS+ has never lost me any data), but as it went to Oracle along with the rest of Sun's assets, I don't think ZFS is really the direction Apple are going to be (or indeed should be) heading in.