Sunday, December 30, 2012

Today, it's my blog's second anniversary, so it looks like this is a good time for some reflection.

2012 in a nutshell

Although I have fewer blog posts this year compared to last year (although the difference is not that big), 2012 was in fact an extremely busy/stressful year. First, I had to write my PhD thesis, which was quite a project. Actually, from February up to August nearly all my time was consumed writing it, when I was not sleeping or eating. Besides my thesis, also some publication work had to be done, such as the HotSWUp publication about deploying mutable software components and a journal version of the WASDeTT paper about Disnix.

Although I was very busy, I've tried to keep blogging as much as possible. In the beginning this worked out well, but later this year I had some difficulties posting anything relevant (as may be observed from April 2012 until October 2012).

I also had some depressing moments. One of them was about science in general. One of the things that made me really mad/frustrated were reviewers "complimenting" our papers with: "this is a remarkable piece of engineering" (which implies that it's not science, so the paper will be rejected). I wrote quite a lengthy blog post about this issue, and dived into literature to find an explanation, which I was able to find.

Although I had some depressing/stressful moments, I also published two blog posts at the beginning of this year, about fun projects involving my good old Amiga, with a bit of overlap to my research. Most of the work for these blog posts was already done last year, but I still had to write them down.

After finishing my thesis draft and a short break, my employment contract with the university expired and I had to switch jobs. The process of finding a new job was quite interesting. A few things that I've noticed is that only a few companies have real interest in who you are and what background you have. A lot of them use a standard application procedures, such as capacity tests, and a lot of "standard" interview questions. Most of them also treated me as an undergrad and offered me salaries way lower than my PhD salary.

Furthermore, before I started talking to these companies I considered myself still as somebody who's more like an engineer as a scientist. But when I spoke to them, I've noticed then I'm more "scientific" than I thought. Although these observations, nearly all the applications I did went fine and I have received many job offers. I was rejected one time, probably because that company was looking for somebody "who just wants to program in Java", instead of a critical person like me.

Finally, I was hired by Conference Compass as a Software Architect. Here, I still use some stuff I was working on during research, such as deployment with Nix and related tools. I've produced two blog posts related to my deployment work there -- deployment of mobile apps on Android and iOS devices.

Blog posts

Another interesting observation is the "popularity" of my blog. This year, I think I have attracted 10 times as many visitors compared to last year. Especially in the last two months, I've attracted a lot of readers, thanks to Reddit. At this moment, the top 10 of the most frequently read blog posts is:

On Nix and GNU Guix. Not so long ago, the GNU project had announced GNU Guix, a package manager based on Nix, implementing an internal DSL using GNU Guile, instead of using the Nix expression language, an external DSL. I wanted to give my opinion about it and the blog post has been picked up by the Reddit community, attracting a huge amount of visitors.

An alternative explanation of the Nix package manager. I wrote this blog post, because I had to explain Nix to my new employer. I've used a different explanation style that is language-oriented and I actually find this version a bit better that the explanation recipe I normally use. I've decided to blog about it and it has been picked up by the Reddit community, attracting a lot of readers.

Second computer. This is the blog post about my good ol' Commodore Amiga. It was also in last year's top 10, and it still seems to be very popular. This shows me that there are still a lot of people out there with nostalgic feelings :-).

An evaluation and comparison of GoboLinux. This blog post was also in last year's top 10 in which I compare the deployment properties of NixOS with GoboLinux. This blog has also been reddited some time ago.

Porting software to AmigaOS (unconventional style). This was a random fun experiment in which I combine some nostalgic Amiga feelings with my current research. I did not expect it to be that interesting, but for some reason it has attracted a lot of readers and even a few people who wanted to try the function I have developed.

Software deployment complexity. One year ago, this was my most popular blog post, that's highly related to my research. It still seems to be relevant, fortunately :-).

Software engineering fractions, collaborations and "The System". This was a blog post I wrote this year in a depressing period, but nonetheless, I still think it's very relevant for anyone involved in the software engineering community to read. Some people say that I'm in a good shape when I'm ranting about something.

The Nix package manager. This was my original explanation of the Nix package manager to use the standard explanation recipe. This blog post was also in last year's top 10.

First blog post. Yet another blog post, that was ranked third in my top 10 last year. People still want to know who I am :-).

Some thoughts

As with my last's year reflection about my blog, I have noticed that my blog attracts many more readers than all my scientific papers combined. Another thing that I observed is that writing for my blog is more convenient than writing papers, because I'm not bound to all kinds of restrictions, such as the page limit, topics that are interesting (and are uninteresting), the amount of "greek symbols" that I should use, whether something is engineering or science etc. It feels like a big relief, it's much more fun, attracts more readers and gives me more feedback.

Actually, it seems that this observation is not unique to me and has been noticed quite some time ago already by Edsger Dijkstra, a very famous computer scientist who published most of his contributions through EWDs, manuscripts which he wrote about anything he wanted. In EWD1000 titled: 'Twenty-eight years', he reflected over his previous 28 research years and one of the interesting things he wrote in that manuscript is:

"If there is one "scientific" discovery I am proud of, it is the discovery of the habit of writing without publication in mind. I experience it as a liberating habit: without it, doing the work becomes one thing and writing it down becomes another one, which is often viewed as an unpleasant burden. When working and writing have merged, that burden has been taken away."

Most manuscripts of him after 1974 were hand-written and he never owned a computer. Because of that people consider him old fashioned. But on the other hand, he was way ahead of his time. In fact, Dijkstra's way of working could be considered an early form of blogging using a pen+paper+photocopier instead of a computer+internet.

I also think that writing more frequently and distributing your findings more often offers much more value. Quite frequently people give the expression that publishing a paper goes like this:

You reach your goal without any obstacles. But in reality it goes more like this:

You will reach many obstacles and you may not even fully reach your goal, or only under special circumstances. These obstacles are also interesting to know about, as well as how to practically implement certain aspects. It gives you better feedback, helps to "structure your mind" and to better reach your audience. Most researchers refrain from doing that.

And another thing is, if you really want any research work to succeed, you much also reach non-researchers. For this practical aspects as well as usable software is important. That's probably a big reason why this blog attract quite some readers.

Concluding remarks

In this blog post I've reflect over 2012 and it seems that it's still worth having this blog. Next year, there is more interesting stuff to come. The final thing I'd like to say is:

Thursday, December 27, 2012

Some time ago, I have explored the mobile deployment space domain a bit, by implementing a function capable of building Android Apps with the Nix package manager. However, Android is definitely not the only platform used in the mobile space. Another major one is Apple's iOS, used by mobile devices, such as the iPhone, iPod, and iPad. In this blog post, I describe how I have implemented a Nix function capable of building Apps for these platforms.

Although it was quite some work to package the Android SDK, it took me less work for iOS. However, all prerequisites for this platform cannot be entirely packaged in Nix, due to the fact that they require Xcode, that must be installed through the App store, and it has all kinds of implicit assumptions on the system (such as location of certain tools and binaries), which prevents all required components to be packaged purely. Therefore, I had to use similar "tricks" that I have used to allow .NET applications to be deployed through the Nix package manager.

Installing Xcode

The first dependency that requires manual installation is Apple's Xcode, providing a set of development tools and an IDE. It must be obtained from the App store through the following URL: https://developer.apple.com/xcode. The App store takes care of the complete installation process.

After installing Xcode, we must also manually install the command-line utilities. This can be done by starting Xcode, picking the preferences option from the 'Xcode' menu bar, selecting the 'Downloads' tab in the window and then by installing the command-line utilities, as shown in the screenshot below:

An Xcode installation includes support for the most recent iOS versions by default. If desired, SDKs for older versions can be installed through the same dialog.

Installing MacPorts

To use the Nix package manager, we can either download a collection of bootstrap binaries or we can compile it from source. I took the latter approach for which it is required to install a number of prerequisites through MacPorts. Installation of MacPorts is straight forward -- it's just downloading the installer from their website (for the right Mac OS X version) and running it.

Installing the Nix package manager

To install Nix from source code, I first had to install a bunch of dependencies through MacPorts. I opened a terminal and I've executed the following commands:

"Packaging Xcode" in Nix

In order to package iOS applications, we have to make the Xcode build infrastructure available to Nix package builds. However, as explained earlier, we cannot package Xcode in Nix, because it must be installed through other means (through Apple's App store) and it has a number of assumptions on locations of utilities on the host system.

Instead we use the same trick that we have used earlier for the .NET framework -- we create a proxy component wrapper that contains symlinks to essential binaries on the host system. Our xcodewrapper package expression looks as follows:

iPhone Simulator. A program to simulate iOS applications on the host system.

Furthermore, the expression contains a check that verifies whether the installed Xcode matches the version that we want to use, ensuring that -- if we upgrade Xcode -- we use the version that we want, and that everything that has Xcode as build-time dependency gets rebuilt, if necessary.

By adding the xcodewrapper to the buildInputs parameter of a package expression, the Xcode build utilities can be found and used.

Building iOS apps for the simulator

Getting Xcode working in Nix is a bit ugly and does not provide full reproducibility as Xcode must be installed manually, but it works. Another important question to answer is, how do we build iOS apps with Nix?

Building iOS apps is pretty straight forward, by running xcodebuild inside an Xcode project directory. The xcodebuild reference served as a good reference for me. The following instruction suffices to build an app for the iPhone simulator:

The above command-line instruction builds an app for the i386 architecture (Apps in the simulator have the same architecture as the host system) using the iPhone 6.0 simulator SDK. The last two parameters ensure that intermediate files generated by the compiler are stored in our temp directory and that the resulting build is stored in our desired prefix, instead of the Xcode project directory.

Building iOS apps for release

Building actual releases for Apple devices is a bit trickier though. In order to deploy an App to a Apple device, a distribution certificate is required as well as an provisioning profile. To test on an Apple device, an ad-hoc provisioning profile is required. For the App store, you need a distribution profile. These files must be obtained from Apple. More information on this can be found here.

Furthermore, in order to use a certificate and provisioning profile, we need to import these into the keychain of the user that executes a build. I use the following instructions to automatically create and unlock a new keychain:

The above instructions generate a keychain with the name mykeychain in the user's home directory: ~/Library/Keychains.

After creating the keychain, we must import our certificate into it. The following instruction imports a key with the name mykey.p12 into the keychain with name: mykeychain and uses the password: secret for the certificate (the -A parameter grants all applications access to the certificate):

$ security import mykey.p12 -k mykeychain -P "secret" -A

We must also make the provisioning profile available. This can be done by extracting the UUID from the mobile provisioning file and to copy the file into ~/Library/MobileDevice/Provisioning Profiles/<UUID>.mobileprovision:

The above command-line invocation is nearly identical to the one that builds an App for the simulator, except that we have a different SDK parameter: iphoneos6.0, and the armv7 architecture. Moreover, we have to specify the identity name of the key, and the locations of the provisioning profiles and keychains.

To test an app on an Apple device, we must generate an IPA file, a Zip archive containing all files belonging to an App. This can be done by running:

By using the Xcode organiser the given App can be uploaded to an Apple device.

In order to release an App through the Apple App store, we must generate an xcarchive, a zip archive that also contains debugging symbols and plist files. An xcarchive can be generated by adding the archive parameter to xcodebuild. The resulting xcarchive is automatically added to the Xcode organiser.

If the build process has been completed, we can eventually remove our keychain:

The above command launches an iPhone instance running our simulator build performed earlier. The former command-line option refers to the executable that we have built. The latter specifies the type of device we want to simulate. Other device options that worked for me are: 'iPad', 'iPad (Retina)', 'iPhone (Retina 3.5-inch)', 'iPhone (Retina 4-inch)'.

The function is very simple -- it invokes the xcodeenv.buidApp function and takes three parameters: name is the name of the xcode project, src is a path referring to the source code and release indicates whether we want to build for release or not. If release is false, then it will build the app for the simulator. It also assumes that the architecture (arch) is i386 and the SDK is the iphonesimulator, which is usually the case. These parameters can be overridden by adding them to the function parameters of the function invocation.

The above expression sets the release parameter to true. In this case, the architecture is assumed to be armv7 and the SDK iphoneos, which ensures that the App is build for the right architecture for iOS devices. For release builds we also need to specify the certificate, provisioning profile, and their relevant properties. Furthermore, we can also specify whether we want to generate an IPA (an archive to deploy an App to a device) or xcarchive (an archive to deploy an App to Apple's App store).

The function invocation above conveniently takes care of the build, key signing and packaging process, although there were two nasty caveats:

To ensure purity, Nix restricts access to the user's home directory by setting the HOME environment variable to /homeless-shelter. For some operations that require a home directory, it suffices to set HOME to a temp directory. This seems not to work for key signing on Mac OS X. Generating a keychain and importing a certificate seems to work fine, but signing does not. I suspect this has something to do with the fact that the keychain is a system service.

I have circumvented this issue by generating a keystore in the real user's home directory, with exactly the same name as the Nix component name that we are currently building, including the unique hash derived from the build inputs. The hash should ensure that the name never interferes with other components. Furthermore, the keychain is always removed after the build, whether it succeeds or not. This solution is a bit nasty and tricky, but it works fine for me and should not affect an existing build, running simultaneously.

Another caveat is the generation of xcarchives. It works fine, but the result is always stored in a global location: ~/Library/Developer/Xcode/Archives that allows the Xcode organiser to find it. It looks like there is no way to change this. Therefore, I don't know how to make the generation of xcarchives pure, although this is a task that only needs to be performed when releasing an App through Apple's App store. This task is typically performed only once in a while.

As with ordinary Nix expressions, we cannot use the previous two expressions to build an App directly, but we also need to compose them by calling them with the right xcodeenv parameter. This is done in the following composition expression:

The above expression calls the xcodeenv.simulateApp function that generates a script launching the iPhone Simulator. The name parameter corresponds to the name of the App, the app parameter refers to a simulator build of an App, an the device parameter specifies the device to simulate, such as an iPhone or iPad.

If we take the composition expression (shown earlier), we also see a simulator attribute: simulate_helloworld_iphone, which we can evaluate to generate a script launching a simulator for the iPhone. By executing the generated script we launch the simulator:

Keychain issues

I have used this function for a quite a few builds and it should work fine. However, in very rare cases it may happen that a generated keychain for signing is not properly removed due to errors in the build infrastructure.

To remedy the problem, I typically open the 'Keychain Access' program and select the following option in the menu: Keychain Access -> Keychain First Aid. In the resulting dialog, I
click on the 'Repair' button. Now all the invalid keychain references are removed and everything should work fine again.

Conclusion

In this blog post, I have described two Nix functions that allow users to automatically build, sign and package iOS applications and to run these in an iPhone simulator instance.

Although the Nix functions seems to work, they are still highly experimental, have some workarounds (such as the key signing) and are not entirely pure, due to the fact that it requires Xcode which must be installed through Apple's App store.

I have conducted these experiments on a Mac OS X 10.7.5 (Lion) machine. Mac OS X 10.8 Mountain Lion, is also reported to work.