Mac OS X 10.4 Tiger

Apple's latest OS X release (10.4) is about to hit the streets. Tiger brings a …

Video in Tiger

Core Video

I left out one of the "Core" technologies when I introduced Core Image earlier: Core Video. That brings the total number of "Core" media technologies to three. As part of the Tiger branding effort, they all have a snazzy new logos that look like glass spheres with gem-like cores.

Core Image

Core Audio

Core Video

I included these icons (and at such a large size) because I think they look cool. Making logos for abstract technologies like Core Image, Audio, and Video is not easy. Whoever created these deserves a raise. They look good enough to eat.

Core Video is a bit of a "secondary" technology, or perhaps a "branding-only" technology. It's essentially the application of Core Image filters to video instead of still images.

That's not to say it isn't impressive. If anything, it highlights the real-time performance of the technology behind Core Image. If you've ever wanted to apply blurs, filters, edge detection, and distortions to live video—and have all of the effects editable in real time—Core Video is the technology for you.

Since Core Video is useless if it does not execute in real time, it requires a programmable GPU. Core Video leverages the GPU via OpenGL, just like Core Image and Quartz Extreme and Quartz 2D Extreme...noticing a trend here?

Applying a series of filters to a still image is one thing. Applying them to video, a series of still images produced on the fly by an entirely separate part of the system, is something else entirely. Not only must Core Video finish processing each frame of video before the next frame needs to be displayed, it must get the frame in a timely manner in the first place to even have a fighting chance.

While this is possible even in earlier versions of Mac OS X, as evidenced by the impressive OpenGL movie player application created by Ars forum member "feelgood," it is much, much easier to do in Tiger using Core Video. That's largely thanks to some significant changes made to the source of most frames of video in Mac OS X: QuickTime.

QuickTime 7

QuickTime 7 is a long-overdue modernization of the oldest continuously compatible digital media system in the industry. Let's start with the changes that make Core Video possible.

It would not be entirely inaccurate to describe the QuickTime-related sessions at Apple's Worldwide Developer Conference over the past four years as a series of decreasingly kludgy examples of how to get QuickTime video to play on a 3D surface.

(Really grizzled veterans of the Apple API wars may recall seeing a purely software-powered demo of QuickTime video playing on all six surfaces of a cube back in the days of "Escher", a.k.a. QuickDraw 3D. Ah, memories...)

In Mac OS X, the "video on a 3D surface" problem has been frustrated by the difficulty of getting each frame of video out of main memory (where QuickTime draws it), converting it into a format that OpenGL will accept as a "texture," and then uploading it to the video card where OpenGL can finally map it onto a surface. Try doing this 30 times each second and you get an idea of the challenge.

QuickTime 7 eliminates the need for kludges entirely. It represents the final unification of QuickTime and OpenGL. Since every other technology that displays or manipulates images is already running on the GPU in Tiger, it only makes sense for QuickTime to move there too. In fact, this change is essential if any other part of the display system wants fast access to QuickTime video frames.

With QuickTime 7 in Tiger, every QuickTime movie is being displayed on an OpenGL surface. It just so happens that the surface is perfectly parallel to the display.

But there's more to QuickTime 7's modernization than just moving the video display onto the GPU via OpenGL. A lot of old, crufty parts of QuickTime's implementation have been superseded by more modern equivalents. Here's what QuickTime looked like in classic Mac OS circa 1995.

QuickTime in 1995

Back then, QuickTime had its own audio and video processing engines. It used QuickDraw to draw to the screen and the Sound Manager to output audio, both of which interfaced directly with the hardware.

QuickTime 7 in Mac OS X 10.4 Tiger looks like this.

QuickTime in 2005

With QuickDraw deprecated, QuickTime 7 uses Core Graphics (another name for Quartz) to draw to the screen. Core Video (and by extension, Core Image) is used to process video. Core Audio finally replaces the horribly outdated Sound Manager. (Though not shown in the diagram, QuickTime 7 still uses plug-ins to support different codecs, just as it always has.)

Core Graphics, Core Image, and Core Video are all built on OpenGL, which in turn interfaces with the video hardware. Core Audio (which I regret that I have not covered in the past, and will not have time to cover here) is built on its own hardware abstraction layer (HAL), with the actual audio hardware below that.

Overall, the QuickTime 7 architecture is more abstracted, more powerful, and more modern. This new architecture made possible another change to QuickTime that was long overdue: the decoupling of video decoding and video display.

When QuickTime 1.0 was introduced in 1991, its video decompression pipeline was based on a simple, and seemingly safe assumption: frames of video would be decompressed in the order that they were to be displayed. This assumption was built into the design and APIs of QuickTime from the very start, and it continued all the way through to QuickTime 6 in 2004. This meant that a compressed frame of video could only depend on earlier frames of video for its successful decompression.

This turns out to be a problem. Modern video codecs (Windows Media 9, Real Video 10, even DivX and friends) often use knowledge of future frames of video to compress a given frame. QuickTime, being a strict "in order" video decompression engine, has absolutely no knowledge of future frames of video when it's decompressing a frame.

QuickTime 7 eliminates this limitation. Finally, the decode order and display order of video frames can be distinct. The most impressive thing about this change is that the full suite of QuickTime editing abilities remain intact. Think about the difficulty of enabling arbitrary cut, copy and paste when a frame of video may depend on future frames for its decompression. What if you cut a portion of a video clip whose final frame depends on a future frame that is beyond the bounds of the cut operation? How can that video be pasted elsewhere safely?

A solution to this problem is "non-trivial," as they say, and it's a big part of why this architecture change has been so long in coming. QuickTime 7 delivers it. To celebrate, Apple includes a modern video codec of their own in QuickTime 7: H.264.

H.264 is the tremendously unglamorous name for the latest video codec in the MPEG-4 standard. It's also known as AVC (Advanced Video Coding) and MPEG-4 Part 10, both of which make H.264 look good in comparison. It's a thoroughly modern codec that rivals the best the competitors (commercial or open source) have to offer. As a part of the widely recognized and supported MPEG-4 standard, it has a leg up on the single-vendor proprietary competitors (WM9, Real) and more corporate weight behind it than most other third-party or open source initiatives (XviD, DivX, 3ivx, etc.)

A rigorous quality and size comparison of these codecs deserves its own article, so I'm not going to go into it now. Suffice it to say that it's comforting to see QuickTime playing in the big leagues again when it comes to video codecs.

QuickTime 7 also includes improved support for multi-track audio, high-resolution audio, media metadata access, and now allows plug-ins to be written using Objective-C.

Speaking of Objective-C, Tiger includes a QuickTime-flavored gift for Cocoa developers. It has historically been, if not exactly difficult, then at least very annoying to use QuickTime from within a Cocoa application. The two technologies came from very different worlds. QuickTime was designed for classic Mac OS (System 6!) circa 1991, while Cocoa is an evolution of NeXT technology.

Tiger finally bridges the two worlds with QTKit. QTKit provides a complete object-oriented interface to all of QuickTime, with Cocoa-style classes for everything: tracks, movies, views, the works.

When Carbon was first being developed, Apple chose to base one of its "important" applications on the new API to prove its usefulness: the Finder. In the software industry, this is poetically called "eating your own dog food." Any long-time Mac OS X user knows how things turned out in the case of the Finder, but the dog food concept itself is a sound one.

In Tiger, Apple has re-implemented the QuickTime Player application using QTKit, and I have to say, the results are immediately impressive. Granted, many of the most impressive features of the Tiger QuickTime Player are a result of the improvements in QuickTime 7 itself; the Player application is simply providing access to them. But at the very least, it's a validation of QTKit's utility.

Here's a brief glimpse of the new QuickTime Player.

The new Cocoa QuickTime player based on QTKit

The player window keeps the same design, but looks subtly different with yet another unique shade of brushed metal. Player windows now resize opaquely, even while the video is playing, though some frames of video are dropped on slower systems.

The "genie effect on playing a movie" animation moves yet another step closer to perfection in Tiger, with the only discernible flaw being the freeze-frame chosen for the initial Dock icon. (Shift-click on the minimize (yellow) widget of a QuickTime Player window while the movie is playing to judge for yourself.)

There's also a new audio/video control panel.

QuickTime Player's audio/video controls

All of the video adjustments work in real time while the movie is playing. Thanks to Core Video, even slow systems don't break a sweat—provided they have a capable GPU, of course. The jog slider also works in real time, scrubbing through the movie at an adjustable speed in either direction. The slider is not as nice as an actual, physical jog wheel, but it is responsive. I only wish the slider was a bit longer to take some pressure off my precision mousing skills.

My favorite new feature is the playback speed adjustment, particularly how it can increase the speed of the audio without changing the pitch. This means I can watch, say, WWDC sessions at 1.5x speed without making every presenter sound like one of the chipmunks. (There is no filter for Bertrand's accent, unfortunately.)

Obnoxious

There's one final unpleasantry to deal with: QuickTime Pro. Like its predecessors, the Tiger QuickTime Player is crippleware. Many of the most useful features are disabled, and can only be enabled by paying Apple $30 for a QuickTime Pro license. The menu items for these "pro-only" features are grayed out and have obnoxious little "PRO" badges next to them.

Don't misunderstand, none of the disabled features in the QuickTime Player are actually disabled in QuickTime 7 itself. They're simply disabled in the player application. Anyone can create a QuickTime player application that goes right ahead and calls the APIs that the QuickTime Player in "non-pro" mode refuses to call. Oh and by the way, the QuickTime browser plugin-suffers from the same intentional feature debilitation.

This has been the case for years with QuickTime, so why get all riled up about it now? Here's why. Mac OS X ships with a complete integrated development environment that supports C, C++, Objective-C, Java, and all of the APIs in Mac OS X (not to mention distributed compiling, a GUI design and layout tool, and a suite of performance monitoring applications). Tiger includes a free web browser, e-mail client, address book, dictionary, thesaurus, font manager, and AIM/Jabber instant message client. When you buy an iMac you get all of the above plus iLife: iPhoto, iMovie, Garage Band, and iDVD.

The total development cost of this software bundle is absolutely huge. The total retail cost of iLife alone is $80. And yet after spending $1,500 or more on a new Mac with this great software bundle, what's waiting for you when you fire it up for the first time and try to watch a QuickTime movie trailer in full-screen mode? Why, it's a nag screen asking you to pay $30 more for the "privilege" of calling the QuickTime APIs that are sitting right there in the library code on your disk.

This is just criminally stupid. It mars the otherwise exemplary out-of-box experience for buyers of consumer Macs especially. Having spent well over $4,000 on my current crop of Mac hardware (plus $80 for iLife '05 plus who knows how much for the Mac OS X Public Beta through Tiger), I find it personally insulting that I'm still not entitled to the "wonders" of QuickTime Pro.

Yeah, sure, I can download a third-party movie player application and find a third-party QuickTime browser plug-in. I can watch movie trailers in iTunes, which will go full-screen even without the magic "pro" key. Or I can google for an illegitimate QuickTime Pro key code. I can even shell out the $30. But it's not the money that bothers me, it's the principle. I'd be happy if Apple simply raised the price of its hardware by $30. On a $4,000+ bill, it's practically a rounding error.

But please, Apple, give up on the QuickTime Pro thing. It's always been annoying, but when viewed alongside today's suite of bundled Apple software, it's downright ridiculous. Worse, it makes the Mac platform look bad when the bundled QuickTime Player application can't do all of the things that make QuickTime so cool: cut, copy, and paste together different kinds of media into a single file, extract and recombine tracks, import and export a huge number of formats, and yes, view video in full-screen mode.

Yeesh. Rant over.

QuickTime 7 summary

Despite the ongoing annoyance of the "QuickTime Pro tax," QuickTime 7 is the most important upgrade to QuickTime in the Mac OS X era. It solves long-standing architectural problems, leverages several of Tiger's other new technologies to do things only dreamt of by QuickTime 6 and earlier, includes its own best-of-breed video codec, and is finally embraced by Cocoa. The new QuickTime Player is good enough to be in danger of reinforcing the (largely uninformed) folk wisdom in the Mac community that rewriting an application in Cocoa automatically makes it better. QuickTime 7 has been a long time in coming, but it has turned out to be well worth the wait.

(And if Apple feels like there's nothing left to do for QuickTime 8 except produce plug-ins for the alphabet soup of audio and video codecs, subtitle tracks, and container formats used by those inscrutable Anime fansubbers, you won't hear me complaining...)

John Siracusa / John Siracusa has a B.S. in Computer Engineering from Boston University. He has been a Mac user since 1984, a Unix geek since 1993, and is a professional web developer and freelance technology writer.