Tuesday, January 28, 2014

Last month, I have been working on quite a lot of things. One of the things I did was improving the Nix function that builds Titanium SDK applications. In fact, it was in Nixpkgs for quite a while already, but I have never written about it on my blog, apart from a brief reference in an earlier blog post about Hydra.

The reason that I have decided to write about this function is because the process of getting Titanium applications deployable with Nix is quite painful (although I have managed to do it) and I want to report about my experiences so that these issues can be hopefully resolved in the future.

Although I have a strong opinion on certain aspects of Titanium, this blog post is not to discuss about the development aspects of the Titanium framework. Instead, the focus is on getting the builds of Titanium apps automated.

What is Titanium SDK?

Titanium is an application framework developed by Appcelerator, which purpose is to enable rapid development of mobile apps for multiple platforms. Currently, Titanium supports iOS, Android, Tizen, Blackberry and mobile web applications.

With Titanium, developers use JavaScript as an implementation language. The JavaScript code is packaged along with the produced app bundles, deployed to an emulator or device and interpreted there. For example, on Android Google's V8 JavaScript runtime is used, and on iOS Apple's JavaScriptCore is used.

Besides using JavaScript code, Titanium also provides an API supporting database access and (fairly) cross platform GUI widgets that have a (sort of) native look on each platform.

Titanium is not a write once run anywhere approach when it comes to cross platform support, but claims that 60-90% of the app code can be reused among platforms.

Finally, the Titanium Studio software distribution is proprietary software, but most of its underlying components (including the Titanium SDK) are free and open-source software available under the Apache Software License. As far as I can see, the Nix function that I wrote does not depend on any proprietary components, besides the Java Development Kit.

Packaging the Titanium CLI

The first thing that needs to be done to automate Titanium builds is being able to build stuff from the command-line. Appcelerator provides a command-line utility (CLI) that is specifically designed for this purpose and is provided as a Node.js package that can be installed through the NPM package manager.

Packaging NPM stuff in Nix is actually quite straight forward and probably the easiest part of getting the builds of Titanium apps automated. Simply adding titanium to the list of node packages (pkgs/top-level/node-packages.json) in Nixpkgs and running npm2nix, a utility developed by Shea Levy that automatically generates Nix expressions for any node package and all their dependencies, did the job for me.

Packaging the Titanium SDK

The next step is packaging the Titanium SDK that contains API libraries, templates and build script plugins for each target platform. The CLI supports multiple SDK versions at the same time and requires at least one version of an SDK installed.

I've obtained an SDK version from Appcelerator's continuous builds page. Since the SDK distributions are ZIP files containing binaries, I have to use the patching/wrapping tricks I have described in a few earlier blog posts again.

The Nix expression I wrote for the SDK basically unzips the 3.2.1 distribution, copies the contents into the Nix store and makes the following changes:

The SDK distribution contains a collection of Python scripts that execute build and debugging tasks. However, to be able to run them in NixOS, the shebangs must be changed so that the Python interpreter can be found:

The SDK contains a subdirectory (mobilesdk/3.2.1.v20140206170116) with a version number and timestamp in it. However, the timestamp is a bit inconvenient, because the Titanium CLI explicitly checks for SDK folders that correspond to a Titanium SDK version number in a Titanium project file (tiapp.xml). Therefore, I strip it out of the directory name to make my life easier:

However, the script throws an exception if it encounters files with timestamps below January 1, 1980, which are not supported by the ZIP file format. This is a problem, because Nix automatically resets timestamps of deployed packages to one second after January 1, 1970 (a.k.a. UNIX-time: 1) to make builds more deterministic. To remedy the issue, I had to modify several pieces of the builder script.

What I basically did to fix this is searching for invocations to ZipFile.write() that adds a file from the filesystem to a zip archive, such as:

The Android builder script (mobilesdk/*/android/builder.py) requires the sqlite3 python module and the Java Development Kit. Since dependencies do not reside in standard locations in Nix, I had to wrap the builder script to allow it to find them:

Although the Nixpkgs collection has a standard function (wrapProgram) to easily wrap executables, I could not use it, because this function turns any executable into a shell script. The Titanium CLI expects that this builder script is a Python script and will fail if there is a shell code around it.

The iOS builder script (mobilesdk/osx/*/iphone/builder.py) invokes ditto to do a recursive copy of a directory hierarchy. However, this executable cannot be found in a Nix builder environment, since the PATH environment variable is set to only the dependencies that are specified. The following command fixes it:

When building IPA files for iOS devices, the Titanium CLI invokes xcodebuild, that in turn invokes the Titanium CLI again. However, it does not seem to propagate all parameters properly, such as the path to the CLI's configuration file. The following modification allows me to set an environment variable called: NIX_TITANIUM_WORKAROUND providing additional parameters to work around it:

Building Titanium Apps

Besides getting the Titanium CLI and SDK packaged in Nix, we must also be able to build Titanium apps. Apps can be built for various target platforms and come in several variants.

For some unknown reason, the Titanium CLI (in contrast to the old Python build script) forces people to login with their Appcelerator account, before any build task can be executed. However, I discovered that after logging in a file is written into the ~/.titanium folder indicating that the system has logged in. I can simulate logins by creating this file myself:

The Titanium SDK also contains a collection of prebuilt modules, such as one to connect to Facebook. To allow them to be found, I run the following command line instruction to adapt the module search path:

I have also noticed that if the SDK version specified in a Titanium project file (tiapp.xml) does not match the version of the installed SDK, the Titanium CLI halts with an exception. Of course, the version number in a project file can be adapted, but it in my opinion, it's more flexible to just be able to take any version. The following instruction replaces the version inside tiapp.xml into something else:

Before the JavaScript files are packaged along with the APK file, they are first passed through Google's Closure Compiler, which performs some static checking, removes dead code, and minifies all the source files.

Building iOS apps from Titanium projects

Apart from Android, we can also build iOS apps from Titanium projects.

I have discovered that while building for iOS, the Titanium CLI invokes xcodebuild which in turn invokes the Titanium CLI again. However, it does not propagate the --config-file parameter, causing it to fail. The earlier hack that I made in the SDK expression with the environment variable can be used to circumvent this:

export NIX_TITANIUM_WORKAROUND="--config-file $TMPDIR/config.json"

After applying the workaround, building an app for the iPhone simulator is straight forward:

After running the above command, the simulator executable is placed into the output Nix store folder. It turns out that the JavaScript files of the project folder are symlinked into the folder of the executable. However, after the build has completed these symlink references will become invalid, because the temp folder has been deleted. To allow the app to find these JavaScript files, I simply copy them along with the executable into the Nix store.

Finally, the most complicated task is producing IPA bundles to deploy an app to a device for testing or to the App Store for distribution.

Like native iOS apps, they must be signed with a certificate and mobile provisioning profile. I used the same trick described in an earlier blog post on building iOS apps with Nix to generate a temporary keychain in the user's home directory for this:

I also discovered that builds fail, because some file (the facebook module) from the SDK cannot be read (Nix makes deployed package read-only). I circumvented this issue by making a copy of the SDK in my temp folder, fixing the file permissions, and configure the Titanium CLI to use the copied SDK instance:

The above command-line invocation minifies the JavaScript code, builds an IPA file with a given certificate, mobile provisioning profile and authentication credentials, and puts the result in the Nix store.

Example: KitchenSink

I have encapsulated all the builds commands shown in the previous section into a Nix function called: titaniumenv.buildApp {}. To test the usefulness of this function, I took KitchenSink, an example app provided by Appcelerator, to show Titanium's abilities. The App can be deployed to all target platforms that the SDK supports.

The above function fetches the KitchenSink example from GitHub and builds it for a given target, such as iphone or android, and supports building a debug version for an emulator/simulator, or a release version for a device or for the Play store/App store.

By invoking the above function as follows, a debug version of the app for Android is produced:

As may be observed, building KitchenSink through Nix is a straight forward process for most targets. However, the target producing an IPA version of KitchenSink that we can deploy to a real device is a bit complicated to use, because of some restrictions made by Apple.

The above commands change the com.appcelerator.kitchensink app id into any other specified string. If this app id is changed to the corresponding id in a mobile provisioning profile, then you should be able to deploy KitchenSink to a real device.

I have added the above renaming procedure to the KitchenSink expression. The following example invocation to the earlier Nix function, shows how we can rename the app's id to: com.example.kitchensink and how to use a certificate and mobile provisioning profile for an exisiting app:

By using the above expressions KitchenSink can be built for both Android and iOS. The left picture above shows what it looks like on iOS, the right picture shows what it looks like on Android.

Discussion

With the Titanium build function described in this blog post, I can automatically build Titanium apps for both iOS and Android using the Nix package manager, although it was quite painful to get it done and tedious to maintain.

What bothers me the most about this process is the fact that Appcelerator has crafted their own custom build tool with lots of complexity (in terms of code size), flaws (e.g. not propagating the CLI's argument properly from xcodebuild) and weird issues (e.g. an odd way of detecting the presence of the JDK, and invoking the highly complicated legacy python scripts), while there are already many more mature build solutions available that can do the same job.

A quick inspection of Titanium CLI's git repository shows me that it consists of 8174 lines of code. However, not all of their build stuff is there. Some common stuff, such as the JDK and Android detection stuff, resides in the node-appc project. Moreover, the build steps are performed by plugin scripts that are distributed with the SDK.

A minor annoyance is that the new Node.js based Titanium CLI requires Oracle's Java Development Kit to make Android builds work, while the old Python based build script worked fine with OpenJDK. I have no idea yet how to fix this. Since we cannot provide a Nix expression that automatically downloads Oracle's JDK that automatically (due to license restrictions), Nix users are forced to manually download and import it into the Nix store first, before any of the Titanium stuff can be built.

So how did I manage to figure all this mess out?

Besides knowing that I have to patch executables, fix shebangs and wrap certain executables, the strace command on Linux helps me out a lot (since it shows me things like files that can not be opened) as well as the fact that Python and Node.js show me error traces with line numbers when something goes wrong so that I can debug easily what's going on.

However, since I also have to do builds on Mac OS X for iOS devices, I observed that there is no strace making ease my pain on that particular operating system. However, I discovered that there is a similar tool called: dtruss, that provides me similar data regarding system calls.

There is one minor annoyance with dtruss -- it requires super-user privileges to work. Fortunately, thanks to this MacWorld article, I can fix this by setting the setuid bit on the dtrace executable:

$ sudo chmod u+s /usr/sbin/dtrace

Now I can conveniently use dtruss in unprivileged build environments on Mac OS X to investigate what's going on.

Availability

The Titanium build environment as well as the KitchenSink example are part of Nixpkgs.

The top-level expression for KitchenSink example as well as the build operations described earlier is located in pkgs/development/mobile/titaniumenv/examples/default.nix. To build a debug version of KitchenSink for Android, you can run:

$ nix-build -A kitchensink_android_debug

The release version can be built by running:

$ nix-build -A kitchensink_android_release

The iPhone simulator version can be built by running:

$ nix-build -A kitchensink_ios_development

Building an IPA is slightly more complicated. You have to provide a certificate and mobile provisioning profile, and some renaming trick settings as parameters to make it work (which should of course match to what's inside the mobile provisioning profile that is actually used):