The AppBundler fork is still maintained but no longer by me. Forward your queries to the capable folks at the Infinite Kind.
Please also note that this article has been last updated on the 29th October 2012 and I'm no longer actively maintaning it. Thanks.

In this guide I'll go through all the steps required to port your Java Swing application to OpenJDK with the goal of preparing it for the submission into the App Store.
Although the App Store guidelines explicitly forbids applications to rely on deprecated or optionally installed technologies (Apple no longer bundles their JDK port so applications can't rely on the user to have it installed), you can still distribute your Java application on the App Store by embedding the OpenJDK 7 OSX port in a native OSX application.

It may sounds obvious but the procedure that follows can be performed only on OSX, not from Windows, Linux or any other OS.

Requirements

OpenJDK

Clone the Mercurial repository(recommended). Compile it with Java 6 or 7 if you already have it. Pick OpenJDK7u to get the stability of OpenJDK7 and the backwards updates from OpenJDK 8.

Download one of the excellent Henry Gomez's pre-built packages.Unfortunately Henry stopped its project but he was kind enough to release all the source code to build the packages.

Don't know which one to choose ?

Getting the source should be the only reasonable option if you're serious about bringing your application to the App Store. However if you want to get started quickly and are not interested in the bleeding edge version, download the Oracle distribution. If you want the latest version of the code but don't want to build it yourself, download one of the Henry Gomez's pre-built images.

(½)The absolute best setup would be to clone the official repo and then replace the jdk subrepo with ourfork which includes more bug fixes for OSX.The situation is a bit different from when I started writing this post. Oracle has integrated all my patches, so unless you require 32bits support the recommended option is to use an OpenJDK binary distribution (and apply the fix for freetype, see below).

AppBundler

Once you have finished downloading the OpenJDK you need to install AppBundler to package your application.

AppBundler is an ant task so just put the downloaded jar in the lib folder in your ant distribution and then follow the instructions to package your application. It still misses a few features like the ability to register your application with a particular file type but it allows to include a Java runtime within the application bundle and building an Info.plist file with most of the required keys.

AppBundler is still missing some basic features like file types association. I recommend instead using our fork.

Note that some keys are compulsory and your application won't be accepted if they're missing so be sure they're present. You must declare at least:

CFBundleIdentifier

CFBundleVersion

NSHumanReadableCopyright

LSApplicationCategoryType

Refer to Apple documentation for the meaning of these keys or just follow this example to be sure they're always included:

<targetname="bundle"depends="..."><!-- Import the AppBundlerTaks from ant lib directory --><taskdefname="bundleapp"classname="com.oracle.appbundler.AppBundlerTask"/><bundleappoutputdirectory="."name="MyApp"displayname="MyApp"identifier="com.foobarbaz.MyApp"shortversion="1.0"icon="MyApp.icns"mainclassname="MyAppMain"copyright="2012 FooBarBaz LLC."applicationCategory="public.app-category.entertainment"><!-- The directory where your OpenJDK runtime is. --><runtimedir="/path/to/openjdk/Contents/Home"/><!-- The bundleapp task doesn't support classpathref so all the run classpath entries must be stated here too. --><classpathfile="MyApp.jar"/><classpathfile="dependencies.jar"/><!-- Workaround since the icon parameter for bundleapp doesn't work. (It's not a bug in AppBundler but in the JavaAppLauncher, see Known Problems). --><optionvalue="-Xdock:icon=Contents/Resources/${bundle.icon}"/><!-- OSX specific options, optional --><optionvalue="-Dapple.laf.useScreenMenuBar=true"/><optionvalue="-Dcom.apple.macos.use-file-dialog-packages=true"/><optionvalue="-Dcom.apple.macos.useScreenMenuBar=true"/><optionvalue="-Dapple.awt.application.name=MyApp"/><optionvalue="-Dcom.apple.smallTabs=true"/><optionvalue="-Xmx1024M"/></bundleapp></target>

If you decide to use my fork, you can do much more:

<targetname="bundle"><taskdefname="bundleapp"classpath="appbundler-1.0ea.jar"classname="com.oracle.appbundler.AppBundlerTask"/><!-- Note the usage of classpathref to avoid copy-pasting all your classpath entries from another target. --><bundleappclasspathref="runclasspathref"name="MyApp"displayname="MyApp"identifier="com.foobarbaz.MyApp"shortversion="1.0"version="build 325"icon="MyApp.icns"mainclassname="MyAppMain"copyright="2012 FooBarBaz LLC."applicationCategory="public.app-category.entertainment"><runtimedir="${runtime}/Contents/Home"/><!-- Specify which architectures you want to support --><archname="x86_64"/><archname="i386"/><!-- Register the application as an editor for PNG and JPG files --><bundledocumentextensions="png,jpg"icon="${bundle.icon}"name="Images"role="editor"></bundledocument><!-- Register the application as a viewer for PDF files --><bundledocumentextensions="pdf"icon="${bundle.icon}"name="PDF files"role="viewer"></bundledocument><!-- Register the application with your custom format, bundled as a package --><bundledocumentextensions="custom"icon="${bundle.icon}"name="Custom data"role="editor"isPackage="true"></bundledocument><!-- Workaround since the icon parameter for bundleapp doesn't work --><optionvalue="-Xdock:icon=Contents/Resources/${bundle.icon}"/><!-- OSX specific options, optional --><optionvalue="-Dapple.laf.useScreenMenuBar=true"/><optionvalue="-Dcom.apple.macos.use-file-dialog-packages=true"/><optionvalue="-Dcom.apple.macos.useScreenMenuBar=true"/><optionvalue="-Dcom.apple.mrj.application.apple.menu.about.name=${bundle.name}"/><optionvalue="-Dcom.apple.smallTabs=true"/><optionvalue="-Xmx1024M"/></bundleapp></target>

See how I used classpathref to declare an external classpath, arch to declare the supported architectures and bundledocument to register the application with particular file types.

Make your application sandbox proof

Starting from the 1st of June all applications submitted to the App Store must be sandbox ready.

For a comprehensive explanation of what the sandbox is refers to the Apple documentation.
In short though, the sandbox is an access control mechanism which restricts interaction of your application with the operating system.

The container is the directory where your application can safely read/write the files required for its correct functioning. It is physically located in ~/Library/Containers/yourapp.bundle.id; I doubt this is likely to change anytime soon so if you have a very simple application you may hardcode this path instead of relying on a native library to return it for you. I say may because I can't possibly know how Apple will react if you hardcode the path of the container instead of obtaining it by using the API.

If your application will require storing user documents in the container or writing temporary files you'll have to acquire the appropriate directories.One route is to write a JNI wrapper and bundle a small native library with your app, another is to use our AppBundler fork which pass the directories to the Java app as environment variables.

If you decide to use our AppBundler, from your Java app you can retrieve the directories using these system properties:

System.getProperty("LibraryDirectory");System.getProperty("DocumentsDirectory");System.getProperty("CachesDirectory");System.getProperty("ApplicationSupportDirectory");System.getProperty("SandboxEnabled");System.getProperty("SandboxEnabled");// (the String "true" or "false")

// If the app is sandboxed:/Users/user/Library/Containers/com.acme.MyApp/Data/Library/Users/user/Library/Containers/com.acme.MyApp/Data/Documents/Users/user/Library/Containers/com.acme.MyApp/Data/Library/ApplicationSupport/Users/user/Library/Containers/com.acme.MyApp/Data/Library/Caches// if the app is not sandboxed/Users/user/Library/Users/user/Documents/Users/user/Library/ApplicationSupport/Users/user/Library/Cachesfalse

Otherwise, if you decide to go with the native route, this is how you would create a native library and acquire the directories where you can write and store app-specific data:

UPDATE: javah generate the header files just fine, the problem was due to the fact I was using Apple's JavaVM framework based on JDK 1.6; instead, compile including the headers of the OpenJDK you're using, for example:

Now that you have your C header you can create a function that will return for example the Application Support folder for your application, which is located inside the appplication container:

JNIEXPORTjstringJNICALLJava_com_foobar_OSXAdapter_getApplicationSupportFolder(JNIEnv*env,jobjectjthis){jstringpath;@autoreleasepool{JNF_COCOA_ENTER(env);NSArray*paths=NSSearchPathForDirectoriesInDomains(NSApplicationSupportDirectory,NSUserDomainMask,YES);NSString*basePath=[pathsobjectAtIndex:0];// Convert the NSString to a jstringconstchar*cString=[basePathUTF8String];path=(*env)->NewStringUTF(env,cString);JNF_COCOA_EXIT(env);}returnpath;}

Migrate existing application data and preferences

If you're not releasing a new application chances is that you are already saving your application data and preferences somewhere, most likely in ~/Library/Application Support/YourApp.

Once your updated application is executed in the sandbox, that folder will be inaccessible for your program, but luckily Apple thought of a migration mechanism, which happens automatically the first time the user start the application. Here's an example of how you would migrate the preferences in ~/Library/Application Support/YourApp to ~/Library/Container/com.yourcompany.YourApp/Data/Library/Application Support/YourApp.

It's not very difficult to understand what's going on in the above example, especially with the comments, but com.apple.security.files.user-selected.read-write deserves a more detailed explanation. The user selected file corresponds to the selected file or directory in a file dialog (either open or save). Note that on OSX this means that you're obliged to use the FileDialog instead of the JFileChooser, this is because the former uses the native NSOpenDialog and NSSaveDialog which are compatible with the sandbox.

Now let's see how to sign your bundle (that you have generated using AppBundler):

I assume that you have downloaded and installed the two keys required for signing the bundle from the Apple developer website. Of course you need to be registered as a Mac developer.

First, you have to sign the application bundle, which you do by executing the following command from a terminal:

All Swing/AWT apps won't even open because they rely on the freetype lib installed on the user system but the sandbox will immediately block your application the access to it and your application will crash.

The proper solution would be to fix the OpenJDK makefiles otherwise a quick workaround is to use two nifty tools called otool and install_name_tools:

First, find out where libfontmanager.dylib,located in the lib directory of your VM, expects to find freetype.

Open a shell and type:

$ otool -L libfontmanager.dylib

Take note of the libfreetype path, likely /usr/X11/lib/libfreetype.6.dylib unless you have built freetype yourself or installed it somewhere else (maybe using MacPorts or a similar tool). Now type:

Type again otool -L libfontmanager.dylib and you should see the change.
We have changed the path to where libfontmanager look for libfreetype to a relative one. So now you have to copy the system or your custom-build freetype library inside the JVM in the same location as libfontmanager.dylib. Relaunch your application, you should at least be able to see it now :)

segmentedTextured buttons not working

I didn't find a solution for this, but it seems that increasing the margin around the button make the problem go away. I couldn't afford to have too much empty space though so I just replaced the decorations with an image.

Dock Icon defaults to Generic Java Application

The (temporary) solution here is to specify both CFBundleIconFile and -Xdock:icon=Contents/Resources/your-icon in your Info.plist file. This could be done easily with AppBundler, I've written an example above.

apple.awt.fileDialogForDirectories property not working.

UPDATE: this patch has been accepted, it's already in OpenJDK 8 and will be soon in OpenJDK 7u

awt.brush.metalLook is not observed.

Sandbox Violation on Runtime Exec

Bug description and ML discussion. Because of the sandbox an application can't create a child process; there are a few workarounds but they all involve writing native code. The real solution is to use posix_spawn.

CA certificates not included in OpenJDK

Private API usage

UPDATE: fixed, this will soon be in JDK7u (hopefully 7u8/10). I updated my repo with the (corrected) patch I proposed.

Our first app submission failed because some awt files use some functions defined in some Apple private frameworks. I replaced the functions I've found with public API and added a patch to the bug report.