Tuesday, February 02, 2010

How to Replace IMAP

IMAP is a complex and difficult protocol. Its original design, RFC 1730, dates back to 1994. Here's a sample IMAP session that shows you all of IMAP's quirks: It's a stateful protocol where you login and select a mailbox to operate on, it has an unconventional parenthesis-heavy representation for structured data, and fetching messages is a multi-step operation.

Design Outline

One of these days, I'll write up a spec for the mail access protocol of my dreams. I'm planning to call it the "reimagined Mail Access Protocol" (short: reMAP) which is handy because reMail already owns the domain name.
I would aim for a RESTful design with the following properties:

All communication over HTTP / HTTPS: Pure TCP connections are great, but for transferring large amounts of email, HTTP is the way to go. Problems like security, parallel downloads, persistent connections, caching, compression, download continuation via ranges, and so on have already been solved. There is no reason to solve them again.

Stateless: There's no reason to introduce state like IMAP does with its selected mailboxes. All you need is a HTTP session cookie used for authentication purposes. Session cookies also allow for things like OAuth. OAuth would let third parties get your permission to access your email without having to give them your username and password.

JSON and UTF-8: All data that's ever sent to or received from the server would be in JSON format. JSON is much more human-readable than XML. UTF-8 would be the only encoding allowed, since it is able to represent any character in the Unicode standard.

Conversations as first-order objects: Gmail, the iPhone SMS app, and Facebook's messaging system have shown the value of viewing messages not individually but in the context of conversations. In reMAP, the server would be responsible for grouping together messages. While you could still access individual emails, the first-order unit of data would be a conversation.

Labels, not folders: Labels are much for flexible than folders. Each conversation should have multiple labels, and the labels would be included when you request the message, rather than having to scan all folders for the message via IMAP.

Stable and unique IDs: IMAP has a UID for each message, but it changes the moment you move the message into a different folder. An IMAP server can also declare all UIDs to be invalid at any moment throughout the session. No more! reMAP would have stable and unique IDs for all conversations, emails, and attachments.

The beginning of the end for MIME: Yes, you could still get the MIME representation of each message that is sent. But MIME is a messy and complex beast. Instead of requesting the MIME-encoded message parts, you could just ask the server to give you the message as represented in plain text or HTML. Attachments can be downloaded in separate HTTP calls.

Push built in: The two prevailing methods for implementing push email are the IMAP IDLE command (not widely available in IMAP servers) and Microsoft's ActiveSync, which requires developers to purchase a license from Microsoft. In reMap, clients could just call an HTTP endpoint on the server which returns as soon as new messages are arrive.

Full-text index on the server: reMAP servers would need to maintain a full-text index of the contents of all messages. There's no reason clients should be required to download and index everything in order to do an exhaustive full-text search of your email.

Enormous Potential

I believe that one of the reasons for the lack of innovation in the email space is the lack of a simple yet powerful email access protocol. Every developer that wants to try something with email needs to first jump through the hoops of IMAP and MIME, or worse, the Outlook Object Model and MAPI. A new protocol like reMAP would lift this burden off their shoulders. We've seen what open, simple standards can do for innovation with Twitter's and Flickr's API. Now imagine unleashing the same sort of creativity to the vast ocean of data that is email. Let me know what you think. My goal with this post was to encourage a discussion about this topic, and your comments are much appreciated. If you liked this, consider following me on Twitter where I often tweet about email-related topics!

Ever since I started dabbling just the user side of IMAP (configuring and using clients on desktop and PDA platforms) I have thought that IMAP is cumbersome. I for one would very much like to see reMAP developed and adopted by some providers, especially Gmail, as Google already uses most of the concepts introduced as part of the reMAP protocol.

Sachin, Voyagerfan - I totally agree that the major mail providers will likely be the agents of change on this. Microsoft Exchange and Yahoo Mail both already have their respective XML-based protocols, but they're not widely supported or offered (for example, Yahoo heavily limits the power of their SOAP protocol for standard accounts). I'm hoping that we'll see these changes happen sooner rather than later. Gabor

It's interested to read this as I had nearly the same idea not long ago. I've implemented a half working prototype in Ruby using webrick and ferret for message indexing. I have it reading a Maildir and it seems to work fine.

I did however go with folders + tagging, although I don't know that the folders give you anything tagging doesnt.

I'd suggest Hessian over JSON. Might as well be efficient in the wire protocol and hessian has a debug mode for when you need to actually read what is going in/out. Hessian also supports RPC which is a nice feature in this case.

You could really accelerate innovation with a change like this. hard thing to know is if it is possible for a small company to pull this off, or if it would require a more community-based approach like OpenID. Eventually they found a friend in Google in their fight against Facebook.

I must say that I agree with most of your points, and I have for long as well thought of an IMAP replacement.

IMAP is the best standard we have available right now but it is both complex to get right as a server and as a client, mostly useless without a bunch of extensions that you're not sure of having on the server, and too much folder centered compared to today's mail volume and usage patterns.

I have a few ideas lying around about this, so if you get any further I'd be happy to contribute !

I think a key would be making the protocol bidirectional, rather than server->client only. That way you could use it both for mail submission and mail reception. That right there would make any MUA using the protocol 50% easier for the average user to set up. (Of course, whether "average users" actually use MUAs instead of webmail anymore is a different story...)

Admittedly you can push messages from the client back to the server with IMAP right now, but it's not normally used for this purpose, which is too bad.

Great article, i also think email is overly complicated and not adapted this web 2.0 era.You are right, the world really needs an easy way to share mailboxes and delegate access effectively, the complexity of today's CRM demonstrates this lack perfectly.

Such a platform would lead to a new wave of CRM and CMS tools based on the content of communications directly instead of having overly complex system that introduce an overhead to manage copies of the actual data beng exchanged.

My thoughts on implementing this:

WebMachine can be used to build the restful framework for push notifications, it is written in erlang which is a great match for this project due to its concurrency model.

The makers of WebMachine, Riak also make a decentralized, fault-tolerant, document oriented database that comes an http interface for json. In my opinion a document oriented database is definitely the way to go for storing email and related data in this context.

Erlang also has the ejabberd jabber server which could be extended to interact with Google Wave and offer a jabber interface.

It could also use the erlang-oauth combined with webmachine to build an interface to expose its email data

This is definitely extremely needed. Email software is stuck in the '80s.

Its a big nightmare to create tools to sync emails. Its probably a good idea to use an XML based interface to store emails. That way, syncing your mails will be as easy as querying XML. Also, allowing easy integration with RSS (which opens up a lot of opportunities in itself).

Stateful protocols make writing clients much easier. And as long as email is still hosted, why bother making things more difficult? I don't see how a stateful protocol makes managing email more difficult. It's a feature, IMO.

The rest however is somewhat agreeable except for JSON. It's still not standardized cannot be validated. Neither can email, but parenthesized structures are easily parseable -- just look at Lisp.

It'll be quite a task to revisit IMAP and see what one could do from a modern context. The best way is to experiment and share your findings. Best of luck!

Parallel downloads is supported but requires multiple connections - but is very, very common. Caching... huh? I'm not sure how that relates to what you're asking. I'd imagine any HTTP email protocol would have an infinite cache life, which is no different than what you'd consider IMAP today.

2) State… do you have all your data on your device, or in the cloud? If you leave nothing on the device, state does become less important - but you'll always have some state on the device even if it's temporary. Managing differences between your current view and what has updated on the server is the whole point of having state.

3) UTF8, no argument. JSON? Ok… it's a manner of talking. That's like saying IMAP is ABNF. You have to define a grammar on top of it, and JSON wouldn't be it.

4) Already in IMAP. See the THREAD command, which takes many arguments for different methods of grouping messages into a conversation.

5) Again, already in IMAP, although it's a bit janky (only certain number of labels allowed).

6) MIME already gives you a stable ID with Message-ID, and good IMAP servers give you a method to search on it. IMAP generally will not invalidate the UIDVALIDITY unless the server itself has been reset (i.e. from backup). Exchange has a similar "reset state" mechanism, it's pretty much required.

7) Huh? You're saying "give me just the html or text". What about all the metadata? From? Subject? etc… where is that coming from now?

And how would you deal with more complex formats? You're reinventing the wheel here for little reason. MIME itself is surprisingly easy to grok. The worst part is all the encoding fun, but that goes away if we all just move to UTF8.

8) Push is built in. Use IMAP IDLE. Done. You can't complain about lack of support here - you're building something fresh, so whatever you make will have even less support. In the future there's also IMAP NOTIFY.

9) And.... again, this already exists in IMAP. Server implementations vary, of course, but many support full text searching.

IMAP isn't perfect by any means, but this list isn't a good reason to ditch it.

I work on a pretty commonly used IMAP server - and I'm sitting on the fence here.

The big pain I see (using offlineimap myself) is that it's:

a) a pain to merge data from two mailboxes without invalidating all the IDsb) a pain to search or move messages between mailboxes.

Basically it's a very one-dimensional index.

Oh, and then there's:

c) you need to fetch ALL FLAGS every time you want to check if anything is changed. There are options (CONDSTORE - which is a bit dodgy across expunges) and QRESYNC which reduce this hurt, but they're not well supported yet.

Also, IMAP plus a random mix of extentions becomes a HUGE compatibility matrix, and you have to offer fallback from all the nice stuff to really shitty workarounds just in case the server doesn't support a feature that makes your life bearable.

So - I'd love to see a decent competitor to IMAP as a protocol. If nothing else because it would reduce the compatibility matrix. Alternatively, I'd be happy for an IMAP5 to be defined which included all the good extentions - so anything which advertised support for IMAP5 had to support them.

Along with a decent test suite a-la DAV's "litmus" this would allow us to move on and simplify a lot of both client AND server code.

I could not agree more, and in fact I've made a couple of attempts along the lines you suggest -- queries done via HTTP requests, replies contain JSON data. If you make sure messages are immutable (except for labels), you also gain benefits of HTTP caching.

One minor thing I want to point out: Unfortunately, the Message-ID header does not give you a good and stable ID since it's not defined on all messages. I know it's supposed to be there but a lot of senders simply don't follow the rules.

I agree with some others and my first choice of protocol wouldn't have been HTTP/REST/JSON, but I don't think whether it is JSON or IMAP/Lisp-like, or XML or binary is really relevant here. What is however is the set of functionality that is accessible though it. Once the features are here, you can easily offer multiple interfaces to it as needed.

Having support for initial mail submission from clients sounds nice (simplifies setup and archiving sent mail), but then still use SMTP to transfert it to other servers as needed.

For having writen an IMAP4 server some while back, I agree with Bron and would welcome this as a potential good IMAP5, without the folder oriented biew but only labels (that can be seen as folders as well when needed). IDLE, THREADS, SEARCH and UID do exist in IMAP4 but they operate on only one folder at time, requiring the client to open many connections for multi-folder operations; How many folders/labels do you have ? That's a huge limitation nowadays.

I wouldn't go for MIME's "Message-ID" either for identifying messages, since there are no guarantees on its unicity at all. A global mailbox UUID would be much better.

With all this, having partial IMAP4 support over the new protocol seems mandatory for easy transition, and as such keeping the original MIME-structure of messages around would be needed.

how about some data integrity too? Im sick of losing my email to these idiotic services. I like how Git uses hashcodes to keep data safer... corruptions and data loss are limited by this pattern. Id love to see stuff like this make into my email which happens to be damn important to keep alive!

In a couple of weeks we are launching our API or Inbox2 which does more or less what you describe here. I will be writing documentation in a few weeks on this. Its basically an http/xml/json based email push (using http long polling) protocol. Authentication is provided by oauth-wrap.

There's a wonderful organization called IETF that would love to hear your proposal. Perhaps now that you're working for a large cash-rich Silicon Valley company, you can use your 20% time to draft a proposal.

The single most important design flaw of IMAP is the omission of first class conversations. This makes IMAP broken beyond repair.Instead of having mailboxes containing messages it should be mailboxes containing threads/conversations containing messages. The THREADS extension to IMAP just doesn't cut it at all. A conversation has to be (potentially) included in more than one mailbox. No way to do that with IMAP properly - e.g. having read status right.

The basics of MIME aren't that complicated, only the I18N stuff (and the awful pgp multipart stuff), which could probably be fixed. And forcing everything to use UTF8 might sound like a good idea in a country that doesn't need it anyway (i.e. one that speaks english), but it may not work for the rest of the world. The basic rules of rfc 822 envelopes and headers are even simpler.

JSON looks like the wrong approach - mail clients already need to deal with rfc822 headers and mime-extensions, and json just looks like a simplified-yet-slightly-different version of exactly the same thing. All you would do is confuse people and then they'd start sending out even more broken email messages than they do already, hard I know as that is to imagine.

MIME will have to stay around for email because HTML cannot be used to encapsulate multi-part content into a single package.

"Pure TCP connections are great, but for transferring large amounts of email, HTTP is the way to go. Problems like security, parallel downloads, persistent connections, caching, compression, download continuation via ranges, and so on have already been solved."

Haha, very funny. The only had to be solved because HTTP is such a horrid beast that turned into something way beyond it's initial imaginings. IMAP hasn't needed to solve most of those problems because they were already part of the protocol. e.g. caching is easier because uid's can be static if the server wants them to be, all connections are persistent anyway, multiple connections are possible, partial down-loads are part of the protocol (by mime-part or byte-range), even pipe-lining of requests is built-in (hides latency without overloading the server) etc. etc. IMAP was designed in an era when slow and unreliable connections were the norm, so it copes with them reasonably well.

And finally your example IMAP session ... well it would be interesting to see just how huge that would be in any implementation of a HTTP-based restful service. Including all of individual http re-connections that might be required, all of the headers passed in each state-less request, and so on and so forth. I think you'll find it will put your idea into a bit more perspective.

Just because *you* don't understand the output doesn't mean it doesn't make a lot of sense. Although the '12 FETCH FULL' command and result for example is the sort of stuff nobody actually uses - since servers never implement it properly.

e.g. just translate this single command into the equivalent restful service including all connections/auth and so on:

And note that the above text is the entirety of the protocol exchange required by IMAP.

Anyway - TBH I think the whole point is moot. IMAP will die a slow natural death simply because the major software vendor in the world hasn't blessed it. Most remote clients in use don't use IMAP because they use some proprietary version of the same, or even embed the client (e.g. gmail). Something like your idea might be another step along that path, but there's probably no point aiming to explicitly 'replace imap' with another hashed up protocol that missed things you didn't realise were needed.

(and i can tell you from experience, a lot of IMAP doesn't make sense until you finally grok it in its entirety, then it is actually a thing of surprising beauty and simplicity with only a few warts that are easily ignored).

About Me

Gabor Cselle
San Francisco, CA

I work at Google, where I'm a Partner at Area 120. Previously, I worked at Twitter on trends, the logged-out homepage, and on MoPub. Twitter acquired our startup Namo Media in June 2014. Before Namo, I was a Product Manager at Google working on Google Now, Android and Gmail. I started reMail, a mobile email startup which was acquired by Google in February 2010.

Before reMail, I was the VP Engineering at Xobni, where we invented a popular plugin for Outlook, and a Software Engineer at Google. I have an MS degree in Computer Science from ETH Zurich in Switzerland.

Views and opinions expressed here are mine and not those of my employer Google/Alphabet.