iCloud Reminders in Org-mode: Talking to OS X with Emacs

In this article I implement a library that allows one to synchronize Org-mode to-do items directly to Apple’s iCal. More generally, I describe a synchronization layer between OS X, Clozure Common Lisp, and Emacs, and muse on its potential.

Background

It is a source of joy and pain to customize and extend Emacs. There are many reasons for the attempt, however, and every person has different reasons. I typically describe a few of mine as follows.

When tasks are performed alongside each other in Emacs, it is conceptually simpler to move between them. Most commands have a sort of “Do What I Mean” universality to them.

Passing data between membranes – copying a URL in a web browser to a chat window, e-mailing a snippet of code to a mailing list, or writing an Org-mode outline before publishing it online – is typically effortless.

Every mode and function in Emacs is another tool in your toolbox. You can easily combine tools in the Unix way to slice through work. When you have more tools at your disposal, you can solve more difficult or nuanced problems.

On the other hand, I have an iPhone and an iCloud account, and don’t spend all my time in Emacs exclusively. I want to be able to use the fantastic software outside of Emacs, but have the power and flexibility of Emacs.

I’ve created a dual-language project. One half is in Emacs Lisp; the other in Common Lisp. It uses the foreign function interface provided by Clozure CL. From Emacs, I use SLIME to access Clozure’s REPL, which allows me to make calls to the storage APIs for iCal. Once I retrieve the results I want, I can pass them back to Emacs. I can also make changes using this API, allowing Emacs to inform iCal of what changes have been made in my to-do lists.

Environment Setup

There is a one-time cost of a complex environment setup. You will need a copy of OS X and Emacs. I am doing all this work on Lion, and I don’t know if earlier versions of OS X will have all the tools we need. I am using Emacs 24:

I should mention that Emacs 24 is incredibly stable. There is a lot of activity and approval surrounding its use as a day-to-day text editor.

Make sure you install Clozure Common Lisp. Clozure’s Objective-C support is a unique feature of its distribution. This article could not have been written without the indispensable support of their wiki and manual.

You can do this however you’d like, but homebrew makes it easy.

$ brew update
$ brew install clozure-cl

You should have ccl64 in /usr/local/bin, and it should be at least version 1.7. Type (quit) at the REPL to exit it.

You will need SLIME, the essential Lisp IDE for Emacs. What I do is open up my Lisp REPL in my shell, install SLIME through Quicklisp, and then symbolically link the package into my Emacs directory (a tool exists to do this, quicklisp-slime-helper, but out of habit I still do it the more old-fashioned way):

Building ffigen

ffigen is a utility that slurps up frameworks and provides foreign function interfaces into ‘intermediate forms’; in this case, Lisp forms. These instructions exist on the Clozure CL wiki. I’m reproducing them here since they’re not easy to find and the wiki may change.

Generating the Foreign Function Interface

Calendar Store is the name of the framework that Apple provides for accessing calendar data. We need to generate its foreign function interface for Clozure. I did so by copying the FFI generation script from the addressbook framework that Clozure provides access to, and turning it into a script for the calendarstore framework. The script, populate.sh, uses the h-to-ffi.sh terminal command installed by building ffigen above.

I should add that this works with the version of Xcode distributed by the App Store, as well as any version of Xcode you may have in /Developer, but I’m only documenting the former. The changes needed for the latter should be self-evident.

Explaining the Bridge

There are a couple aspects of Objective-C and how they translate through the Bridge that we need to be aware of.

Methods in Objective-C

Objective-C uses a message passing syntax. Whereas languages like C++ bind method names to some code during compilation, Objective-C resolves message receivers during runtime. Message names are typically longer than method names because they also contain identifying information about the message’s parameters, separated by colons.

Whereas in C++ you may call something like:

obj->method(foo, bar)

In Objective-C you would call:

[obj method:foo andBar:bar]

And we’d consider the message name to be method:andBar:. Yes, even the colons are part of the message name. In our bridge, that method invocation would become:

(#/method:andBar: obj foo bar)

Now that we know this, we can start using Apple’s API to pull objects out of the runtime. Open up a REPL with Clozure CL as your inferior Lisp. First we need to load up the bridge.

CL-USER> (require 'objc-support)

Many commonly used Foundation Framework objects are available for us immediately. These objects start with the prefix ns-. An NSDate would become an ns-date. An NSString would become an ns-string. Say we wanted to create an NSDate with the current date and time, and print its string representation. Let’s put this functionality in a function called today_as_string.

Take a look at the difference as to how methods are called in Objective-C. today_as_string, implemented in Objective-C, and then Lisp.

Memory Management

Based on the Clozure documentation (§15.3.3), an autorelease pool is created for us in the toplevel. This means as good citizens, we should manually release the memory when we finish using it. Calling [NSDate date] automatically calls init and alloc on the instance, and the (#/release today) call above dovetails that by decreasing the reference count.

We are given a reader macro for generating NSStrings (prefix it with a pound sign, #), and we can get a Lisp string out of an NSString with Clozure’s lisp-string-from-nsstring function:

CL-USER> (ccl::lisp-string-from-nsstring #@"test")
"test"

The use of the pound sign in the above code ((#/date ns:ns-date)) is a reader macro as well. You can get a hint as to how it works by checking the contents of the package nextstep-functions.

Introducing nsclasp

I’ve written a small utility library for Common Lisp called nsclasp. It provides a set of functions that are very useful when working with Apple’s Foundation Framework. You can retrieve it by cloning it into your Quicklisp installation’s local-projects directory and refreshing the project list:

Our handling of the NSArray is a little clumsy. We can’t just mapcar over it, we need to explicitly iterate over its elements by index. That’s why nsclasp is helpful, as it contains a method to make this a little nicer:

But no matter how you do it, you should have been able to retrieve all your calendar names. As it turns out, this is an important piece of information to have, as the Foundation Framework doesn’t yet allow us to create the kind of calendar that would sync with iCloud for us. For the time being, we have to create the calendar we want our TODOs synced to, in iCal.

First Steps: Creating a CalTask

Now let’s combine this on the tail end of creating an iCal reminder, and sending it to a calendar.

Tasks exist in iCal as instances of the CalTask object of the CalendarStore Framework. Every task has a title, UID, a parent calendar, alarms, due dates, completed dates, notes, and other such attributes. We have to describe a mapping. Our input function takes an Emacs buffer and returns a list of parsed TODOs, and what calendar they should be synced to.

I run the function on it, and after getting asked which calendar I want the items to belong to, I get the following list of lists.

(icalsync-el-transform-todos "example.org")

:buffer

/…/example.org

:point

78

:calendar-uid

C854CB60…

:incomplete-p

t

:title

Fix toilet

:scheduled

nil

:uid

nil

:dt

nil

:buffer

/…/example.org

:point

60

:calendar-uid

C854CB60…

:incomplete-p

t

:title

Trash

:scheduled

nil

:uid

nil

:dt

nil

:buffer

/…/example.org

:point

17

:calendar-uid

C854CB60…

:incomplete-p

t

:title

Recycling

:scheduled

(0 0 0 15 4 2012 0 t -14400)

:uid

nil

:dt

nil

When we upload a task into the Calendar Store, we’re going to provide its title, deadline, and whether or not it’s complete. There’s also priority, which would be nice, but not necessary for the time being. Once the upload is successful, we’re going to also retrieve the task’s UID, and its dateStamp property, and save them in the property drawer of the TODO, in order to make synchronization easier.

From here, all we need to do is run the Emacs Lisp functions needed to transform all the TODOs and upload them into the Calendar Store. Keep iCal open when you do this and make sure the Reminder list is visible.

First Steps: Retrieving Changes

So now you have your Org-mode tasks in a Reminder list on your iPhone. Brilliant! You wander around the town, happily marking off the items you’ve completed. How can we make Org-mode aware of the items that were checked off?

This is a very rudimentary method, but it works. What we’ll do is update TODOs based on buffer, the same way we uploaded them to iCal. For a given buffer, we’ll walk through the TODOs, seeing if they have a UID handed out by the calendar store. If so, we ask the calendar store about it. We compare the time-stamp of the TODO to the time-stamp of the reminder, and if the reminder has been changed more recently, we update the TODO.

Next Steps

There are a few things missing from this implementation. We don’t check for the existence of a task before creating it from the corresponding TODO. Priority is not synced. Org-mode is not made aware of items added to Reminders.app. We track where our TODOs are based on the buffer name and their position in the buffer, which will quite easily break if we shuffle our TODOs around. Only the reminder’s completion status is synced back to Emacs, but if its description or due date changes, we should synchronize those as well. As a proof-of-concept, however, I think this is a solid start.

In addition, the behavior that you use to turn Org-mode files into lists of reminders – or vice versa – is entirely up to you. There are countless ways to do so. A conversion could, for example, take each calendar, and make it a headline in an Org file, with each task becoming a checklist item with a property drawer. Another conversion could check the TODOs for a custom condition, i.e., only tell iCal about TODOs which have a certain property in their drawer, or are scheduled after a certain date.

All the code written for this article is available in my elstar GitHub repository. The elstar name for my repository is a bannerhead of sorts, an invitation for this method to be explored further, and to collect the results in a shared place. I welcome any contributions.

Conclusion

The point of this article was not just to create a synchronization layer for iCal to Org-mode, but to explore the method of synchronization itself. For Emacs users on OS X, the potential exists to fill in the gaps with their usage of operating system (and associated mobile/cloud convenience) and text editor. Some other ideas that may be worth pursuing:

An Emacs mode for interfacing with iTunes

Functionality to convert Address Book entries into BBDB or org-contacts entries

Turning events on calendars into Emacs diary entries

I look forward to seeing what some enterprising Lisp hackers can come up with.