Main navigation

Last Week on My Mac: The extent of APFS’s problems

Every Mac user should be very interested in what happens with Apple’s new file system APFS, because it is the strongest indicator of Apple’s commitment to the platform.

Currently, every iOS device running 10.3 and later is running APFS, seemingly fairly happily. In fact, in the vast majority of cases, so happily that the user isn’t even aware that its file system has changed.

In contrast, only a small minority of Macs start up from an APFS drive: those which have been upgraded to (or shipped with) High Sierra, and boot up from an SSD. If you’re one of the millions whose Mac starts up from a Fusion Drive, an Apple-only product which has been pushed very hard as an ideal compromise between the speed of an SSD and the low-cost high-capacity storage of a hard disk, then you have either not upgraded yet, or are running High Sierra on Apple’s old HFS+.

In theory, in a few weeks or months time, a High Sierra update will bring APFS to Fusion Drives, and possibly conventional rotating hard disks (that seems less certain), and we’ll all move on, upgrade to High Sierra, and live happily ever after. If you’re confident that will happen, then you should listen carefully to Tim Standing, in his presentation at the MacSysAdmin conference in early October.

Tim – an immensely knowledgeable and experienced Mac software engineer, who for more than twenty years has been half of SoftRAID – draws attention to one of the adverse effects of copy-on-write, perhaps the single most important technology behind APFS. Copy-on-write is the heart of snapshots in APFS, its support for versioning, even the increased metadata protection which makes journalling unnecessary.

I have previously shown how copy-on-write works in the context of a single edit, and versioning. Let me illustrate its downside the same way.

Let’s start with a single continuous file on our storage. The ‘chunk’ which this occupies (shown in green) is known as its extent. Now, edit it in three sections.

That single extent is changed into a total of seven extents which together make up the file. If the file is being versioned, the three brown areas in the original extent would remain part of the previous version: we’ll ignore that here. Now, let’s edit the file a second time, changing three sections of its data again, two from the original extent and one from the previous edit.

After just two editing sessions, our single-extent original file has fragmented into no less than thirteen extents.

On an SSD, this poses no particular problem, although the file system is going to have to keep track of a very large number of extents. This makes APFS significantly more computationally intensive than HFS+, but then the older file system was designed for 68K Classic Macs with their pedestrian processors.

On a rotating disk, as Tim Standing says, everything starts coming to a grinding halt. To play that movie or audio, macOS needs to stream it continuously, which means seeking to each of those thirteen extents. Multiply that through the many files in macOS which your Mac needs to access, and you can see why the performance of APFS on rotating disks is far inferior to that of HFS+.

That is, though, something of a worst case. There are at least four solutions which Apple will be considering.

Tim Standing is of the opinion that Apple will end up layering APFS with CoreStorage, a project which he thinks could be completed quickly and simply. Another solution would be to turn copy-on-write off when APFS accesses hard disks, something Apple has apparently claimed it could do, but hasn’t done so yet.

The classic answer to file fragmentation on hard disks is to continuously defragment, which we are used to on HFS+. Whether that would prove feasible for the continuous and extreme fragmentation resulting from copy-on-write is unclear, though.

The fourth potential solution lies in the fact that copy-on-write is not an all-or-nothing technique, but can be implemented with different degrees of granularity. APFS decides how finely to fragment files, and the larger the extents used, the less the dangers of severe fragmentation.

By requiring larger extents to be involved in copy-on-write, APFS could, for example, limit the number of fragments resulting from the example editing process to seven rather than thirteen. The trade-off is that the benefits are also reduced. Perhaps Apple’s engineers are currently performing this sort of tuning to arrive at the best compromise for rotating disks.

Little of this has any bearing on APFS for iOS, but is crucial to the future of macOS. Like it or not, many Mac users are going to relying on rotating hard disks for years to come. They may not boot from them, but Time Machine backups and other mass storage systems reaching beyond a few terabytes will remain of great importance for the foreseeable future. And even as I write this, Apple is still selling 2017 iMacs with hard drives, and Fusion Drives.

In the coming weeks and months, Apple will demonstrate its commitment to macOS in APFS. If we’re left waiting and wondering, we’ll know that the Mac’s days as a serious computer platform are numbered, and Apple will be content to sell pretty laptops as iPhone accessories.

18Comments

Another very insightful, helpful article on Apple’s APFS – thank you so much for all you do for keeping the Apple Community informed and up-to-date. After watching Tim Standing’s excellent presentation, do you have an “idea” of what the third bug he is working with Apple on to get “fixed”? And are you a beta user of SMART Alec, and if so, do you suggest joining the beta program or wait for the APP to be launched in the Spring 2018?

Thank you.
I have no idea of the third bug: Tim knows what he is talking about, with respect to file systems and HFS+ in particular, but I don’t have any idea. Perhaps that’s as well!
I haven’t tried SMART Alec out yet, but suspect that it is excellent, even as a beta. I am going to try it out when I get a chance, and think that it would be good to join the programme.
Howard.

The kind of file modification that Tim is talking about is pretty rare in day to day use. Most high-level APIs such as NSDocument strongly encourage the entire file to be written in one shot. And even if you did have a file open and write tiny chunks here and there, almost all files are going to be memory-mapped. It will be up to the VM to decide when, where, and how those extents get written to disk.

Thank you.
I think the best answer at present is that, in the absence of documentation for APFS and tools to look at extents and fragmentation, we simply don’t know how bad this might be, but unless copy-on-write is turned off, additional fragmentation will be inevitable.
I would be very suspicious about high-level interfaces which think that they’re writing the entire file as a single, new extent. From the limited documentation which we do have, we know that editing a cloned file doesn’t work that way, as APFS seems to determine the delta to be written. So, according to Apple, editing clones will result in fragmentation in most cases.
We also know that there will be similar effects when you make snapshots, again implemented within APFS and invisible at any higher level. As snapshots are the basis for Mobile Time Machine, this implies that anyone using that feature will incur fragmentation as a result of the way in which APFS manages snapshots.
We don’t know how APFS manages document versioning, but the implication is that (unless an app manages this itself at low level) versioning may well utilise cloning, which will in turn lead to fragmentation.
The fourth way in which this will necessarily occur is in the volume metadata, which use copy-on-write to allow rollback, and avoid any need for journalling to ensure metadata integrity.
So even if plain file saves avoid the perils of fragmentation, there are four ways in which fragmentation seems very likely if not inevitable. Performance figures shown for APFS on rotating disks appear to bear this out, as does the continuing delay in the release of a version of APFS which can be supported on Fusion Drives and rotating disks. Interestingly, each of these issues looks primarily to be a problem for macOS and not iOS, underlining my argument.
Howard.

The high level APIs certainly don’t know anything about extents. But they definitely write an entire file as one NSData object. When using these preferred, high-level APIs, there are never any edits. All writes take place by creating an entirely new file, using a single NSData, inside a new temporary directory. Once the write is successful then the old file is swapped out for the new and the temporary directory removed. That is how NSDocument “edits” a file. APFS does have special logic to swap out directories and I’m sure that was implemented precisely for this reason.

It is possible directly open files and actually edit them. Doing that in the context of a modern, document-based app is not for the faint of heart. I would be interested to know how apps that use traditional UNIX I/O, such as databases, will work with APFS. These programs may have files open for a long time with many small writes. Will each write go into a new extent? Or will a still-open file write into existing extents? This is the kind of question that could benefit from some documentation.

Are you saying that Apple’s documentation of clone editing and versioning is not correct – that APFS actually rewrites the entire clone, and a complete new file for each version?
If so, that knocks a large whole in its claimed features.
Howard.

No. I’m sure what little Apple has documented is probably correct. I’m just saying that there are a number of different ways to write data depending on the API being used. The preferred API writes an entire file in one, more-or-less-atomic operation. Plus, that file is memory-mapped. Any discussion of what APFS might or might not do is academic. Unless one uses the lowest-level POSIX APIs, it is all opaque and beyond direct control.

Also, these kinds of operations make me think about the new File Coordination APIs. When they were first released, only about 20% of the documented methods worked. I assume that percentage is higher now, but I haven’t checked. I know that Apple has made this API a highly critical component. But the point is, just because something is documented doesn’t mean it exists. Documentation can be, as they say on the financial statements, “forward looking”.

I think he’s likely correct that Fusion Drives will be an overwhelming demand from Apple Marketing people and that Core Storage will be the response. I would be more comfortable with that than more changes to APFS (changes the filing system in the last beta, a couple of weeks before it shipped – seriously?!).

Thank you.
I don’t think that this is just a marketing issue: it is a matter of co-ordination of corporate policy.
For part of Apple to have promoted its Fusion Drives, and still to feature them in current products, other parts of Apple, including file system engineering, must be co-ordinated. To develop any file system without regard to the hardware which is going to use it is frankly crazy.
Maybe the APFS team’s sole mission was to develop a file system for SSDs. If that’s the case, it is Apple admitting that it doesn’t really have any interest in its computer products, or their users. I find that more than a little incredible – I hope!
Howard.

Well, I just had an interesting experience. I have gotten on an encryption kick. Having done this on my laptop, I am presently encrypting the drives on my cheese grater Mac Pro, which has High Sierra on SSD (now my main drive), as well as on the stock HDD. Another internal HDD still has Sierra. The SSD, of course, is now APFS.

After encrypting the SSD, I turned my attention to my venerable LaCie 1TB Firewire drive. I had to initialize the drive to do this, changing the partition scheme from Apple to GUID (I used Coriolis iPartition to do this), before encryption was possible. Again, it’s an old drive.

I was expecting a CoreStorage HFS+ drive as a result, but instead the encryption process silently converted the hard drive to APFS. Had I Googled my intentions in advance, I would have found some user experiences like this. This should not be an issue with the other internal drives, as I shall be booting into them and using FileVault (and, again, one of them is plain old Sierra anyway.) It seems that this will just happen when one encrypts non-bootable drives, or at least drives that are not at present the startup disk.

I am presently copying back in the files that were on that drive, that were worth keeping. I guess this drive will be a guinea pig for APFS on HDD. Should be interesting.

Thank you.
Yes, I have heard of others who opted to encrypt hard disks only to find them being formatted in APFS. I thought that Apple might have addressed that issue by now, but seemingly not in 10.13.2.
Another issue that I heard others complain about was interrupting encryption. Before HS, you could shut down etc. in the midst of File Vault encrypting, and when you next started up it would just carry on where it had left off. Early releases of HS apparently couldn’t do that, and tended to trash the part-encrypted drive. I think that must have been fixed now.
I hope that it works out OK for you.
Howard.

It’s not too big a deal on this drive. If I don’t like it, I can probably just boot into my Sierra drive to repeat the procedure and get the expected result. I’ll only have the hassle of moving those several-hundred gigs of files back and forth again. 🙂

On the subject of interrupted encrypting (which is still in process as I write), I have in fact rebooted the machine and everything seems nominal.

Lots of great info, but I dispute your conclusion. I am convinced that the Mac will remain a serious computing platform. Apple’s renewed commitment to the “pro” market, as demonstrated with its iMac Pro offering and the “Apple-forecasted” Mac Pro, should serve as ample evidence.

Given that copy on write (COW) is at the core of ZFS, the best industrial strength file system currently available, I’m guessing that any problems being seen with it are due to Apple’s implementation and not COW itself. Apple made a huge mistake by not going with ZFS as their file system.

Honestly if we project on what will happen with Apple, you have to consider that SJ understood the world of Unix, understood Sun Microsystems, NeXT and Silicon Graphics, Inc. He saw enterprise computing from these vendors and pushed for 64 bit, and probably understood the potential of 128 bit XFS. SJ was looking to build a better UNIX platform for the masses and had the lessons and experience of using UNIX in the past.
From this he brought UNIX to the platform and was constantly using his experience and knowledge of both developers and big enterprise systems to Apple.
Tim Cook is clearly not this person. Cook came from a background of Just in Time supply chain and logistics / vendor management and manufacturing. Again, I would argue that SJ trumped himself by hiring the wrong guy for the long term. If I had access to Tim’s brain, I would immediately ask him to be humble in terms of making bold technical decisions that he isn’t qualified to make – and let the people who are passionate about the existing REBUILT platform continue the tradition of the market SJ already rebuilt.
Astoundingly in the case of self driving cars, TC wasn’t thinking about things like the entire road infrastructure of United States would have to be transformed at a cost of TRILLIONS of dollars before this is viable when building out trains is more of a valid opportunity. They could improve upon train technology with HyperLoop R&D people. I would also express that Social Justice is not for the workplace as much as it is for his private time. He can easily start a foundation and push for his perspective and interests in his private foundation. Social Justice and equity of quotas for upper management and board members is harmful to the business SJ rebuilt. Apple isn’t an opportunity for TC’s ego, but for SJ’s legacy of product. TC could also take a page from history and like Google, split the company into two – an R&D group like Bell Labs or Xerox PARC or even Alphabet, and leave the main company alone and not subject to non-sense like if Apple has the right number of women to men ratios according to HIM (not even working with sociological statistics on a scientific level). Apple is a technology company and not a social justice platform for his self-guided passions. He must answer to the share holders and their expectation of what this company must maintain. If SJ learned anything, it was removing the self and ego from Apple products and development and the market that this schema adopted expects that to continue.
From the evidence of the departures of the original ‘rebuild’ engineering team slowly leaving, I can only guess it is impossible to get into TC’s head to have him maintain what their market wants. TC is turning this into his company, and hence you see things in the market that are not based on experience and clarity of thought: APFS prematurely releasing to market. Mac Mini no longer modifiable, UI in iPhone being cluttered with majorly adjusted layout, AppStore offering InApp purchases with the same name, but at different costs… etc. All they need to do is reference what SJ did upon his return and get back to that, but he will not. Maybe TC will step down soon?

Many interesting thoughts, thank you.
One thing I would take issue over, though, is ZFS. At the time I was a keen proponent. Looking back, it would have been disastrous for iOS, tvOS, and watchOS, which would in turn be left with plain old HFS+ still.
I think that APFS is too device-based and not as complete a file system as ZFS would have been, but across all of Apple’s products it is a far better compromise. Even on a MacBook Pro, ZFS would have been a beast to get the best from.
Howard.