More

iCloud sync speculation

Here’s my last minute technical speculation what iCloud will be in terms of sync :-)

It’ll be sync-enabled WebDAV on a large scale.

I spent the last 10 years working on synchronisation, in particular SyncML. SyncML is an open standard for synchronisation, created in 2000 by the then big players in the mobile phone industry together with some well known software companies.

SyncML remained a niche from a user perspective, despite the fact that almost every featurephone built in the last 9 years has SyncML built-in. And despite the fact that Steve Jobs himself pointed out Apples support for SyncML when he introduced iSync in July 2002 at Macworld NY.

As we have learnt by now, iSync (and with it, SyncML for the Apple universe) will be history with Lion. And featurephones are pretty much history as well, superseded by smartphones.

Unlike featurephones, smartphones never had SyncML built-in (a fact that allowed me earn my living by writing SyncML clients for these smartphone platforms…). The reason probably was that the vendors of the dominant smartphone operating systems, Palm and later Microsoft, already had their own, proprietary sync technologies in place (HotSync, ActiveSync). Only Symbian was committed to SyncML, but forced by the market share of ActiveSync-enabled enterprise software (Exchange) in 2005 they also licensed ActiveSync from Microsoft.

So did Apple for the iPhone. So did Google for Google calendar mobile sync. Third party vendors of collaboration server solutions did the same.

For a while, it seemed that the sync battle was won by ActiveSync. And by other proprietary protocols for other kinds of syncing, like dropbox for files, Google for docs, and a myriad of small “cloud” enabled apps which all do their own homebrew syncing.

Not a pleasant sight for someone like me who believes that seamless and standards based sync is as basic for mobile computing as IP connectivity was for the internet.

However, in parallel another standard for interconnecting calendars (not exactly syncing, see below) grew – CalDAV. CalDAV is an extension of WebDAV, which adds calendar specific queries and other functionality to WebDAV. And WebDAV is a mature and widely deployed extension of HTTP to allow clients not only reading from a web server, but also writing to it. Apple is a strong supporter of WebDAV since many years (iDisk is WebDAV storage), and is also a driving force behind CalDAV. Mac OS 10.5 Leopard and iOS 3.0 support CalDAV. And more recently, Apple implemented CardDAV in iOS 4.0 and proposed it as an internet draft to IETF, to support contact information the same way as CalDAV does for calendar entries.

This is all long and well known, and CalDAV is already widely used by many calendaring solutions.

There’s one not-so-well-known puzzle piece however. I stumbled upon it a few month ago because I am generally interested in sync related stuff. But only now I realized it might be the rosetta stone for making iCloud. I did some extra googling today and found some clues that fit too nicely to be pure coincidence.

The puzzle piece is this: An IETF draft called “Collection synchronisation for WebDAV” [Update – by March 2012 it has become RFC6578]. The problem with WebDAV (and CalDAV, CardDAV) is that it is was designed as an access method, but not a sync method. While it is well possible to sync data via WebDAV, it does not scale well with large sync sets, because a client needs to browse through all the information available first just to detect the changes. With a large sync sets with possibly many hundred thousand files (think of your home folder) that’s simply not working. The proposed extension fixes exactly this problem, and makes WebDAV and its derivates ready for efficient sync of arbitrarily huge sync sets, by making the server itself keep track of changes and report them to interested clients.

With this, a WebDAV based sync infrastructure reaching from small items like contacts and calendar entries to large documents and files (hello dropbox!) is perfectly feasible. Now why should iCloud be that infrastructure? That’s where I started googling today for this blog entry.

I knew that the “Collection synchronisation for WebDAV” proposal was coming from Apple. But before I didn’t pay attention to who was the author. I did now – it’s Cyrus Daboo, who spent a lot of time writing Mulberry, an email client dedicated to make best possible use of the IMAP standard. Although usually seen as just another email protocol, IMAP is very much about synchronisation at a very complex level (because emails can be huge, and partial sync of items, as well as moving them wildly around within folder hierarchies must be handled efficiently), so Cyrus is certainly a true sync expert, with a lot of real-world experience. He joined Apple in 2006. Google reveals that he worked on the Calendar Server (part of Mac OS X server supporting CalDAV and CardDAV), and also contributed to other WebDAV related enhancements. It doesn’t seem likely to me they hired him (or he would let them hire him) just for polishing the calendar server a bit…

Related to the imminent release of iCloud, I found a few events interesting: MobileMe users had to migrate to a new CalDAV based Calendar by May 11th, 2011. And just a month earlier, Cyrus issued the “WebDAV sync informal last call” before submitting the “Collection synchronisation for WebDAV” to IETF, and noted that there are “already several client and server implementations of this draft now”. And did you notice how the iOS iWork apps just got kind of a document manager with folders? After becoming WebDAV aware only a few months ago?

So what I guess we’ll see today:

a framework in both iOS5 and Mac OS X Lion which nicely wraps WebDAV+”Collection synchronisation for WebDAV” in a way that makes permanent incremental syncing for all sort of data a basic service of the OS every app can make use of.

a cloud based WebDAV+Sync storage – the iCloud

a home based WebDAV+Sync storage – new TimeCapsules and maybe AirPorts

and of course a lot of Apple Magic around all this. Like Back-to-my-Mac and FaceTime are clever mash-ups of many existing internet standards to make them work “magically”, there will be certainly more to iCloud than just a WebDAV login (let alone all the digital media locker functionality many expect).