The iPhone SDK: APIs Apple Didn't Want You to Know About

Author's Note: Special thanks to Jay Freeman (Saurik) who has done remarkable work in updating the open source tool chain to support iPhone v2.0

With the release of Apple's SDK for building iPhone applications, many have plunged head-first into this new platform for the first time, with the new-found excitement that comes in discovering something entirely new and innovative. The energy surrounding the iPhone has been building steadily since its release last June, and Apple's initial "beta" offering of their SDK gave developers many of the tools they needed to get engaged. Within a short time, however, the community hit a brick wall in many respects, leaving many disenchanted by the restrictions imposed on developers. While Apple insists that the SDK provides the same tools used to create their own software, developers have found that they don't have access to the same low-level functions of the iPhone, such as the ability to run applications in the background, build certain types of objects, or use low-level frameworks such as CoreSurface, Celestial, or LayerKit — all of which provide direct access to graphics and sound components. These, along with many other features, are found in Apple's own applications, but nowhere to be found in the SDK.

A little bit of history is required to fully appreciate this debacle. Since June 2007, the open source community had been hacking on the iPhone and developed a huge audience (estimated to be around 40% of the market). Shortly thereafter, a group of hackers managed to free the iPhone's operating system to run open source software and had succeeded in building their own community SDK (software development kit) for compiling third-party iPhone applications. Since then, the development community surrounding the iPhone has grown considerably, and hundreds of great applications have been made freely available for the device via a popular community software installer. The "Installer" application serves as a kind of "online library," allowing anyone using it to download and install programs directly from the iPhone (over the air), without iTunes or even a desktop machine.

By fall 2007, the rest of the iPhone community had tuned in. Nicholas Penree, author of the popular jailbreakme.com website, had reported that over one million iPhone users had used the site to enable their device for third-party applications. Since then, it has been estimated that over two million iPhone users are now running the third party "Installer" application, giving them full access to the entire public software library for the iPhone. Open source development has become so popular, in fact, that O'Reilly has recently commissioned me to write, and published a book titled "iPhone Open Application Development," which documents many of these APIs and teaches developers how to write applications for the iPhone.

Jump ahead to March 2008. Apple finally realized what a huge financial opportunity they were missing out on when they snubbed third party developers, and decided to release their own version of what the community already had been using for nearly a year, a software development kit (the Apple SDK) and application distribution chain (the iTunes AppStore). Ironically, due to this delay, Apple was surprisingly the one lagging behind the open community, and rather than the open source community duplicating commercial efforts, Apple embarrassingly became the one trying to duplicate the open source community today.

With the introduction of the Apple SDK, developers gauged its functionality based on a comparison to the unofficial, open source SDK released last August. In the process of building this custom, open source compiler for the iPhone, the development community exposed the many low-level APIs (application programming interfaces) available on the device. Using tools such as class-dump, nm, and just plain old trial-and-error gave developers access to the full breadth of functionality available deep within the iPhone's frameworks. It was used to write applications that could look and act just like Apple's preloaded software, so when Apple announced that their SDK was "the same set of tools," many expected that it would look and feel like the open tool chain. Very few had anticipated the many restrictions they've come to find in the official SDK. While roughly 75% of the two SDKs do overlap, the remaining 25% has shown to be very restrictive, removing the developer's ability to do "the real fun stuff" with their application.

In this clear and concise book, Jonathan Zdziarski -- one of the original hackers of the iPhone -- explains how developers can design third-party software that will run on this device. You'll learn about iPhone's proprietary development environment, the Objective-C language it uses, and background on the operating system. You also get detailed recipes and working examples for several iPhone features.

Back to the present, the APIs available in the Apple SDK are useful for building your average game, or your average application, but very lacking for building applications with more sophisticated, low-level requirements. Fortunately, there is another set of interfaces that Apple never wanted you to know about, the "real" set of APIs that Apple uses. These are the same interfaces that have been made available through the unofficial SDK "tool chain," only we didn't know it at the time. The great news is that you can use the Apple SDK (via Xcode) to build applications using these hidden APIs, and this article will show you how.

It's important to note that it is unclear whether using these hidden APIs will disqualify your project from being listed in Apple's AppStore. When it comes down to it, the issue is not a technical matter, but rather a licensing and policy issue. Using these APIs can help extend your application's functionality, but be warned that this may also mean you'll need to distribute your application using the community "Installer," or on your own. The good news is that this is what many developers have resolved to do for the sake of writing better software, and with a market penetration of over 40%, the community installer is able to reach a very large audience. It is also believed that the iTunes AppStore will not be available to iPhone users who have unlocked their phones and are running on unauthorized networks, further expanding the potential of the community software installer.

Now for the fun part, for those who are, or will soon be running the Aspen (iPhone OS) software on their iPhone, and would like to build applications using these hidden APIs, there are now two different methods you can use: the open source tool chain or Apple's official SDK (with some customizations). Using either, it is possible to not only write applications that take full advantage of the low-level frameworks used by Apple's own software, but also to build existing applications written for the open source tool chain.

Using Private APIs in the Apple SDK

Apple's low-level APIs, made available in the open tool chain, contain some objects and methods that have been intentionally left out of the Apple SDK, to help control what developers can and can't do. If you think about the ramifications of what it would mean to be able to write your own movie player, for example, it would be very unwise for Apple to foster applications that competed with their own.

Introducing our first example of a missing framework, the CoreSurface framework. CoreSurface allows for direct writes to a screen surface, making applications such as custom movie players and software emulators possible. The framework itself is included with the iPhone and the Apple SDK, as it is used by higher-level libraries such as OpenGL, but developers cannot take advantage of it because the framework headers are missing.

Another example is in a missing set of objects named UIPreferencesTable, used for building settings screens inside an application. Apple's document insists that you create a preference bundle for your application, so that users have to exit your program and use the "Settings" application. This all caters to look-and-feel, but some find it unreasonable, and equate removing this object as removing the ability for a desktop program to have a "Preferences" menu option.

To take advantage of these hidden APIs within the Apple SDK, you'll need to install the header files for these objects and then modify your Xcode projects to use them. An unauthorized set of private headers is being maintained by a group of software developers named "the iPhone Dev Team." Unofficial, these headers are used by the open source SDK. To download and install these low-level APIs, do the following from a Terminal window on your desktop:

This will install the APIs into a new directory named /Developer/SDKs/iPhoneOS.sdk/Versions/iPhoneOS2.0.sdk. Now you'll need to instruct your Xcode project to use them instead of the "official" APIs. To do this, follow the steps below:

Go to the Project menu and select Edit Project Settings

click on the Build tab at the top

Scroll down to the section titled GCC-4.0 - Language

For the option titled Other C Flags, enter the following values, all on one line (remove carriage returns), and then save your settings.

If you build your application, you'll now be using the low-level set of APIs in addition to the standard set. Because the low-level APIs include many private frameworks, such as CoreSurface, you can now import these frameworks into your application.

To use such a framework in Xcode, you'll first need to switch your build to "Device - iPhone OS 2.0". The reason for this is that the simulator does not include many of the frameworks included on the iPhone. In fact, the simulator is unable to even run some "official" SDK applications, such as those using OpenGL. You'll only be able to build and test these low-level frameworks on the iPhone itself.

Right-click on the Frameworks folder in your project and select Add -> Existing Frameworks. This will drop you into a frameworks folder, where you can select one or more frameworks to add. You may need to navigate to the PrivateFrameworks folder, one directory up from Frameworks. If you have trouble finding it, start from your hard drive and navigate to /Developer/iPhoneOS.platform/Developer/SDKs/iPhoneOS2.0.sdk/System/library/PrivateFrameworks/

Choose the framework you want to add, such as CoreSurface.framework, and click Add. The framework will then be added to your Xcode project.

Now that the framework is linked in, add the appropriate includes to your project, for example:

#import <CoreSurface/CoreSurface.h>

Now you're ready to go! Many examples of how to use the many private APIs and frameworks can be found at http://www.iphonedevdocs.com/, as well as in my O'Reilly book. You will write more functional code and sleep better at night knowing that your application isn't restricted by a device manufacturer's policies form-factor requirements!