Tech —

Mac OS X 10.4 Tiger

Apple's latest OS X release (10.4) is about to hit the streets. Tiger brings a …

Core Image

Core Image is Tiger's new image processing framework. It joins Core Audio, Apple's existing audio processing framework. The technologies share more than just a naming convention. They both provide a plug-in architecture for processing their respective media types. These plug-ins are called "units." Core Audio uses Audio Units and Core Image uses Image Units.

Core Image is conceptually very simple. An input image is passed through a series of chained Image Units, resulting in an output image.

Core Image

The entire image pipeline uses 32-bit floating-point values for each of the four channels (red, green, blue, and alpha) in order to preserve as much accuracy as possible across many calculations. Tiger ships with 60 predefined Image Units covering most common image manipulation tasks: blurs, distortions, color adjustments, etc.

Image processing doesn't get much more straight-forward than the diagram above. But it's in the implementation where Core Image really shines. Two aspects of Core Image make it a lot more attractive than the diagram above might suggest.

Although Core Image is conceptually a chain, with the output one one Image Unit becoming the input for the next, a naive implementation of this design would be prohibitively slow. The performance would degrade roughly linearly as the number of Image Units increased, and there would be a tremendous amount of i/o associated with reading and writing all the intermediate results.

A naive implementation of Core Image

This is not how Core Image is actually implemented in Tiger. Instead of running through each Image Unit in sequence, producing an intermediate image after each step, Core Image uses what's called "lazy evaluation."

Passing an image through an Image Unit doesn't actually cause anything to be rendered. The output of an Image Unit is simply a "recipe" for how to produce the desired result. As the image passes through all of the Image Units, each Image Units appends more steps to the recipe.

Actual rendering is only done when absolutely necessary. Usually, that's at the end of the sequence of Image Units. At that point, the entire recipe is actually run, producing the final result image. So the implementation looks a lot more like this.

Core Image in Tiger

There's just one "do work" stage, no (potentially large) intermediate results to read and write, and a single "effective Image Unit" (the big cloud in the diagram) that's combination of all of the actual Image Units.

The second clever aspect of Tiger's Core Image implementation is how it produces and runs the code that makes up the "effective Image Unit." Core Image includes a just-in-time (JIT) compiler that converts the implementation-agnostic "recipe" for the entire Image Unit chain into optimized code, targeting the fastest hardware available at the time of execution.

In many cases, that will mean AltiVec. On multi-processor Macs, the Core Image JIT compiler will even create SMP-aware code that utilizes all available processors. Since this code is created on-the-fly and is tailored to the specific set of Image Units that contribute to the overall recipe, the potential for optimization is limited only by the intelligence of the JIT compiler.

If a reasonably modern video card is installed, AltiVec may not be the "fastest hardware available." Instead, the Core Image JIT compiler may target the GPU, producing (essentially) a pixel shader to run the recipe via OpenGL. This is right in the GPU's sweet spot, so the job gets done fast.

How fast? The standard Core Image demo shows a handful of Image Units being applied to a source image, each one executing instantly as it is applied. Once the entire recipe is complete, any one of the Image Units can be adjusted—increasing the radius of a blur or moving the center of a distortion effect—and the image will update in real time. It's impressive.

Even more impressive is this: adding Image Units seems to have no affect on performance. Well, at least initially. So long as the Core Image JIT compiler produces code that can execute comfortably on the current GPU, there will be no discernible performance difference between a chain of two Image Units and a chain of eight. But once the recipe requires more than the number of rendering passes that the GPU can execute in, say, 1/30th of a second, performance starts to slow perceptibly.

Anyone who sees a Core Image demo can't help but think of Photoshop. Core Image seems to do the same thing Photoshop does with its filters, only Core Image does it in real time. Visions of a "Photoshop killer" powered by Core Image quickly spring to mind.

Unfortunately, this vision suffers from a very narrow view of what Photoshop actually does. Although Adobe Photoshop was initially (and understandably) identified with its image filtering features when it was introduced in 1990, they are perhaps the least interesting features of the application today.

It's not very difficult for a wannabe "Photoshop killer" application to re-implement any one of Photoshop's filters. (If that was all it took to be a Photoshop killer, the (free) GIMP would be the dominant graphics application today.) The hard part is matching Photoshop's fifteen years of refinement in the areas of drawing and editing tools, file format compatibility and conversion, workflow integration, and all the other things that keep Photoshop customers coming back version after version.

That's not to say that creating a Photoshop killer is impossible, or that it can't or won't incorporate a technology like Core Image. But Core Image doesn't really help the effort much. In fact, as a Mac-only technology, any real Photoshop killer would probably be forced to forego Core Image entirely.

Delusions of grandeur aside, Core Image really is powerful and I'm sure there are plenty of non-Photoshop-killer applications that will benefit from using it. In fact, Tiger actually uses it itself to implement this next technology.

Share this story

John Siracusa
John Siracusa has a B.S. in Computer Engineering from Boston University. He has been a Mac user since 1984, a Unix geek since 1993, and is a professional web developer and freelance technology writer. Emailsiracusa@arstechnica.com//Twitter@siracusa