Mac OS X 10.5 Leopard: the Ars Technica review

After a two-and-a-half-year wait, Leopard is here. John Siracusa covers the …

The Kernel

Let's go in the opposite direction entirely and dive into the core of the OS. We'll work our way back up to the higher levels eventually. For now, it's kernel time.

There was a bit of a kerfuffle about the future of the Mac OS X kernel back in the summer of 2006, fed mostly (as these things often are) by an information vacuum. The summary is that Apple wasn't releasing the source code to its then-new x86 kernel (as it had with all previous kernels) and wouldn't say why. Many theories sprang up to fill the void. I, of course, had my own pet theory.

The most logical reason that I can think of for Apple's refusal (thus far) to publish the source code to the x86 version of the Mac OS X kernel is that the kernel shipping today in the x86 versions of Mac OS X Tiger is an evolutionary dead end, and therefore not worth the effort to pretty up and publish.

Presumably, all of the major work on Mac OS X, the kernel or otherwise, has long been focused on Leopard. Now imagine that the Leopard kernel has significantly diverged from the Tiger kernel. Maybe it's a new kernel entirely, or maybe it has significant changes to support virtualization more efficiently, or something in between. Apple seems to be holding its cards close to its chest until WWDC. In the meantime, pushing out the source to a soon-to-be defunct incarnation of the Tiger kernel might not be along the critical path.

I'll be very surprised if there's no big kernel-related technology or announcement at WWDC. That said, I don't see any pressing need for major kernel shenanigans in Leopard, just more of the same kinds of improvements that came in Tiger. Maybe no big announcement really would be the best possible outcome.

WWDC came and went (and came and went) and there was no major kernel announcement. The Leopard kernel does indeed contain "more of the same kinds of improvements" that we've seen over the life of Mac OS X, and that's not a bad thing at all. (I'll save my kernel pipe dreams for Mac OS X 11.0, I suppose.)

A lot of the kernel rumor craziness had its origins in the idea that Mac OS X is a poor performer at the kernel level due to some fundamental design choices. This is a long-standing meme with some sound (though often misapplied) computer science theory surrounding it, as well as the expected bevy of dubious benchmarks.

As usual, the truth is much less dramatic. The core OS team at Apple is, perhaps predictably, the polar opposite of the graphical design team. Apple's kernel engineers in particular are pragmatic, cautious, and wise. They're also human, however—if you prick them, do they not bleed?—which may explain why they spent some time at WWDC spelling out the philosophy behind Mac OS X's kernel development process.

Apple's focus is on system-level performance, not micro-benchmarks. The kernel team's job is to make the software at the higher levels look good. If improving the performance of some tiny aspect of the kernel tenfold does not provide a measurable performance increase for some user-visible feature or task, it's not an effective use of development time, benchmark bragging rights be damned.

That's not to say that Apple's kernel team isn't competitive. But when it comes to dedicated kernel benchmarks, there's a natural home-field advantage: Linux tends to do well on LMBench, Solaris does well on libmicro, and so on. This is not surprising; the choice of benchmark determines where optimization is done. Apple's decision to measure kernel performance "from the top" by looking at the behavior of the real applications running on the full OS dictates which aspects of the kernel get the most attention.

In Mac OS X in general, and in Leopard in particular, improvements to scheduling and latency are important. There's a big difference between being "fast" and being "responsive," and Apple's focus is on the latter. Here are a few of the highlights from the Leopard kernel. (For the nitty gritty details, there's always the source code... or will be, once Apple updates its repository.)

Kernel highlights

The Leopard kernel is better about scheduling processes on CPUs, moving from a single, flat queue of processes to a hierarchical one that better reflects the actual hardware (e.g., two separate chips, each with two CPU cores). Bouncing a process from one CPU to another is bad for performance; the on-chip caches don't get a chance to properly warm up. But multiple cores on the same chip often share some caches. A hierarchy of process queues in the kernel created with this knowledge allows for better scheduling choices.

The Leopard virtual memory system is better about determining which pieces of memory are actually being used by your application right now and which are safe to swap out to disk. When it comes time to swap to disk, Leopard will (finally!) dynamically allocate swap files, which means that you should get some disk space back when the memory pressure abates.

Resource limits, the bane of my existence in Tiger and earlier, are dynamic where possible in Leopard. These are things like the number of open files or processes per user and so on. If you've never bumped up against these limits, consider yourself lucky; a lot of things stop working in very bad ways when you can't open any more files, for example.

I routinely run into these limits in Tiger and have often been forced to take heroic measures to increase them. A few of the defaults have also increased in Leopard (e.g., the default maximum number of processes per user has increased from 100 to 266. I'll still keep mine over 2000, thanks). And for good measure, there are even a few new limits on previously unlimited resources like resident and wired memory sizes.

The Leopard kernel also has a new "sandboxing" system which forces certain processes to run in their own isolated, restricted environments for security reasons. Apple's implementation is based on mandatory access control (yet another "MAC" acronym that's not short for "Macintosh"). These sandboxes are defined, in typically unpretentious Unix style, by plain text files (examples can be found in /usr/share/sandbox) and are applied to many system services in Leopard, including Bonjour, Quick Look, Spotlight, NTP, and many others.

DTrace

Perhaps the most significant change in the Leopard kernel is the addition of DTrace. DTrace was developed by Sun and is open source. Apple's core OS team has continued its streak of shrewdly identifying and adopting best-of-breed open-source projects and has completed the substantial task of porting DTrace from Solaris to the Mac OS X kernel. DTrace solves a long-standing kernel development problem, and does so in such a fantastic way that it creates new opportunities for Apple to help all programmers, not just kernel hackers.

To understand how DTrace helps kernel developers, consider the following scenario. Let's say you're a developer working on some aspect of process creation in the kernel. To help during your development, you'd like some sort of notification every time a new process is created. So you find the function in the kernel that creates a new process, and you add a bit of your own code to the beginning of that function that prints some information to a log file. Now you recompile your kernel, reboot, and continue your work.

Unfortunately, you've hard-coded at least three things using this technique: 1) the fact that you want some debugging information, 2) the location of the inquiry, and 3) the mechanism of the report. Furthermore, it's likely that you'll want similar bits of debugging code in other places in the kernel in the future, and it's unlikely that you'll want every one of these bits active at the same time.

So, being the good little programmer that you are, you come up with a more general solution. At each point where some debugging code may be useful, you wrap the code in a conditional expression that asks, "Should this piece of debugging be turned on right now?"

This seems like a good solution until you've filled the kernel with these snippets. Remember that the kernel, by its nature, tends to contain code that executes very quickly and very frequently. A tiny check to see if a particular piece of debugging should be turned on may only take a millisecond, but if the entire routine executed in ten milliseconds before you added this check, you've just increased the execution time significantly. And if this routine is called many thousands of times per second, you're starting to talk some real wall-clock time down the drain. Now multiply this by many hundreds or thousands of debugging probes in the kernel, and it becomes clear why all these checks cannot be left in the finished product.

The obvious solution is to convert these debugging checks from conditions that are evaluated at runtime to conditionally compiled code. When debugging is enabled during the kernel build process, some or all of the debugging code is included in the build. But when debugging is disabled for a production build, the debugging code is omitted entirely from the kernel.

Though I've simplified things greatly, this is the gist of traditional kernel-level debugging probes. You work with a special "debug build" that may be slow but which contains all the diagnostics you need for development. When you need to add, enable, or disable some debugging code, you recompile and reboot. When you're happy, you compile an optimized production build of the kernel that contains none of this debugging code.

Into this environment comes DTrace, which proposes the following seemingly impossible combination of features.

No recompilation required. Enable or disable debugging probes in real time on a running kernel.

Near-zero overhead when not in use. The impact of disabled debugging code is so small that all such code can be left in production kernel builds.

Programmers reading this will be forgiven for cringing a bit at the smell of self-modifying code, but my advice is to just close your eyes and think of England. The bottom line is that it actually works, and works well.

DTrace supports its own simplified programming language called "D" (no, not that one) which is used to define probes. Here's an example that prints a notification every time a new process is created.

In action, it's indistinguishable from magic. You write these little text files with script-like bang-pound lines using this weird C-like language and you have essentially free reign to grope all over the kernel. (You have to be root to run DTrace at all, for obvious reasons.)

The D language does not support branching, subroutines, or loops—a good thing, because accidentally creating an infinite loop or recursion inside the kernel definitely should not be one tiny plain-text script away. You also can't use DTrace to modify kernel memory or CPU registers or to call arbitrary functions.

But within its limited scope, D is still quite powerful. It supports most common C/C++ data types, aggregates, local variables, and a whole slew of shell/awk-style conventions: script arguments in $1 .. $N, BEGIN and END blocks, etc. It even has native support for cute ASCII histograms. It's quite pleasant to use—especially compared to recompiling the kernel and rebooting.

And remember this is all running on a plain old consumer copy of Leopard, not a special build. DTrace is included on all Leopard systems; it's not an optional install. This means that developers can rely on their users having it. Since DTrace scripts are plain text files, remotely debugging a thorny problem by e-mail suddenly got about a thousand times easier.

(Debug kernel builds that contain a full complement of symbols and other metadata are still useful. DTrace does not replace them. What it does do is provide an unprecedented level of flexibility on top of them—flexibility that remains even in the shipping version of the kernel.)

Xray Instruments

Install the developer tools, and you'll get a Garage Band-like GUI application for applying debugging instruments (get it?) to individual applications or the entire system. This application was called Xray for most of its development life, which explains the icon. It's now called Instruments for reasons that surely involve lawyers. If you'll forgive me, I'm going to keep calling it Xray for the rest of this review.

Unsurprisingly, many of the most powerful instruments are based on DTrace. There's even a GUI for creating custom DTrace-based instruments, plus the ability to record and play back a series of actions. Mmm... automated GUI-based performance regression testing.

DTrace and Xray invite good questions. "How many files does my application open on launch?" "How many times is a particular function called?" "What does the memory usage of my application look like over time?" DTrace and Xray make the previously daunting task of answering these questions almost trivial and (dare I say it) fun. I can't imagine any Mac developer seeing Xray and not instantly longing to sic it on his application.

All of this newfound power can't help but lead to better, faster, more stable applications—from third-party developers as well as from Apple itself. And it's all thanks to an obscure, open-source, low-level kernel debugging framework from Sun.

State of the kernel

With Tiger, Apple finally completed the kernel's transition from its NeXT roots to its Mac OS X future by nailing down the kernel APIs and providing a clear path forward. Leopard has taken the first big step down that path. The addition of DTrace is the most significant change. It's an entirely new feature and was not created with Mac OS X's kernel in mind. DTrace will also have the biggest impact on the development process and by extension on the nature and quality of applications available to users.

The rest of the changes are "more of the same," and that's a good thing: performance optimizations, scalability improvements, better standards compliance, all in appropriately conservative doses. The addition of DTrace must have helped a bit with the rest of Leopard's development, but it has taken a while for DTrace to come up to speed on Mac OS X. The real payoff will come in the next major version of the OS, which will have spent its entire development life in a post-DTrace world.

John Siracusa / John Siracusa has a B.S. in Computer Engineering from Boston University. He has been a Mac user since 1984, a Unix geek since 1993, and is a professional web developer and freelance technology writer.