JMAP! Does it work?

If you've been a regular follower of the FastMail blog, you've heard of
JMAP. If you haven't, go check out that link. You'll get
to see Neil and Bron telling you why it's cool. If you're really into the
idea, you might also like reading the spec, which
is quite easy to read and presented in soothing FastMail Blue.

If you can't be bothered, though, it's like this: JMAP is a way of accessing
your email in the cloud. It's like IMAP, but better. (It's often joked that
JMAP is one better than IMAP, but that's not quite right. Then it would just
be IMAQ. JMAP is 1,000 better than IMAP.)

JMAP looks really good in theory, but until somebody implements it, it's not
all that useful. Fortunately, FastMail and others are hard at work
implementing it. That means there are a number of partly-implemented JMAP
servers, meaning that there are also a bunch of only partially-satisified JMAP
consumers, like me. I'm quite pleased to count myself among the earliest users
of JMAP, because it means I'll end up with the most stories about the ways in
which early implementations failed horribly. To help gather those stories,
I've been working on the beginnings of a generic test suite for JMAP
compliance, to check any given JMAP server against a rigorous set of torture
tests. (This also ends up helping JMAP implementors make sure they've gotten
it right, and I guess that's why FastMail is paying me to do it, but it's
certainly not the part that makes it fun for me!)

Writing the JMAP test suite has required writing a number of new pieces of
software, and I'm going to give you a quick tour through them. If the idea of
seeing still-half-baked Perl 5 code fills you with an indescribable sense of
dread, maybe you should stop here and go read about
revolutionary mail technology of the future (of the past)
instead.

Perl is Funny

So, I write a lot of Perl, and it's a great language, and I wouldn't want it to
get the wrong idea about how much I love it, but it's got a bit of a huge blind
spot. It's just terrible at telling numbers and strings apart. Given these
two variables in Perl...

my $x = 5 ;
my $y = "5";

...Perl is pretty much not going to be able to tell you that they're not
identical. Sure, there are some tricks that work some of the time, but there's
no really great solution. This is fine as long as your program doesn't talk to
anybody else's programs, but when you're testing a network server,
program-to-program interaction is rather the order of the day.

JMAP uses JSON for data interchange and JSON has a strict
distinction between types. These two hunks of JSON are quite different:

{ "mailboxId": 123 }
{ "mailboxId": "123" }

If your server hands back the wrong 123, your client is going to have a bad
day. This meant we had two options: write our JMAP test system in something
other than Perl or add a number/string distinction to Perl. We decided to do
the latter. See, Perl? That's how much we love you. This got us
JSON::Typist, a type annotator for
JSON data. When applied to a freshly JSON-decoded hunk of data, it indelibly
marks numbers and strings as such:

This lays the foundation for testing the types of JSON data in a fairly
typeless language. The next layer is
Test::Deep::JType, which extends
Perl 5's excellent Test::Deep system to
account for maybe-typed data. You can write:

...and assert that when the JMAP server says it has updated message 123, it has
returned the number 123, not the string. This is vital, not only because
downstream clients will be confused by the wrong type, but also because Neil
will give you a withering look once he realizes how you've screwed up. He might
forgive you, though, if you do penance by implementing a test framework to
prevent future mistakes of that sort. Here's hoping, anyway, right?

Enough Perl, Let's JMAP

Now that we can actually inspect JMAP responses to make sure that we've gotten
the right things, the next thing we want is to actually get some JMAP
responses! For this, we need a JMAP client. JMAP clients can come in many
levels of complexity — one of JMAP's strengths! Everybody's favorite command
line browser, curl, can be an adequate client for some things. For users who
just want to read their mail, FastMail's in-browser JMAP client is a much more
satisfying experience. You want the right tool for the job, and the right tool
for testing JMAP is JMAP::Tester.

JMAP::Tester is an HTTP client build to do only four kinds of things:
authenticate with a JMAP server, upload and download binary content to it, and
execute JMAP methods against it. All of these are trivial operations where
curl would really suffice, but JMAP::Tester provides rich result objects that
make it easy to inspect responses for just what you expect without flailing
about every time. For example:

We can make a request (ensuring we send the right types of data), assert that
we got exactly one line of response, which is some kind of setFoos call,
assert that our server state changed, and that we got back a record with an
id string. Meanwhile, if we fail a critical assertion, like getting a pies
response rather than an error, the entire subtest will cleanly abort and move
on to the next subtest.

Testing JMAP Mail

Some of you may now be wondering why there's now delicious Pennsylvanian
molasses pie smeared all over your shiny new mail protocol. Well, JMAP isn't
just for mail. It's also a generic client/server protocol that can be used for
all sorts of things, including cloud-based bakeries. Most of the excitement
about JMAP, though, is about JMAP Mail, which deals in messages and mailboxes
more than cakes and pies.

This is where we (finally!!) reach the still very much under-construction
JMAP::TestSuite, which layers JMAP
Mail helpers on top of JMAP::Tester:

This is the fundamental unit of testing in the JMAP test suite: one hunk of
code creating, updating, or destroying batches of data and then inspecting the
results with Perl's best-in-class testing libraries. Here, we create a batch
of mailboxes. Each one is associated with the id (x, y, or z) by which its
result can be found, which mirrors the way that creations and results are
correlated in JMAP itself. With this system, any time a bug is found in a JMAP
implementation, it should be trivial to safely add a new test for that bug and
run it against every server.

In the next few iterations of the test suite, we'll be adding generators for
valid-but-random test data and improved debugging output so that "just add
print" for debugging can be done automatically. This helps prevent the
ever-embarrassing git commit, "remove 692 data dumps accidentally left in after
debugging," which appears eleven times in my commit history for the last week.

Pointing the Suite at your Server

Different JMAP servers have different capabilities. JMAP specifies an
authentication mechanism, but not every server uses it. JMAP doesn't specify
how to create new users, so different servers will have different mechanisms
for creating test users. (Or maybe, too, you want to re-use a test user over
and over.) To abstract all this away, the JMAP test suite offers the idea of a
"server adapter" that handles authenticating a JMAP::Tester to a server. We've
got adapters for Cyrus, "standard JMAP," and the JMAP Proxy.
To point the whole test suite at
your server, you just tell it which adapter to use:

...and look at that! A bug in Cyrus,
discovered by the
JMAP test suite! I look forward to finding ever more and weirder bugs in JMAP
implementations through the perpetual expansion of the test suite… but also to
implementations made increasingly robust by the test suite. Even better for
the authors of JMAP servers, they can run the test suite in the privacy of their
own offices and, unlike me, it won't giggle maniacally at each new bug.
There are still a few features left to add to the test
suite, and a lot of tests left to write, but it's a promising beginning.

Thanks for reading the FastMail Blog: news, views and the occasional rant from the FastMail team. Want to keep up to date? Subscribe to our RSS feed, or sign up below and weʼll email you every time we post something new.