Do we need WADL?

Everybody's atwitter about WADL, a description file for REST services, and since
it's supposed to be RESTful I regularly get questioned about it.
For this post I'm going to experiment and adopt Stephen O'Grady's
Q&A style.

My thanks to Patrick Mueller
and Dave Johnson for volunteering to be
guinea pigs for my arguments, and thanks to #redmonk for the use of their channel where we initially held the discussion.

Q: Does REST need a description document?

A: No.

Q: You're new to this Q&A format, aren't you? How about giving some more details.

A: There are, as always, two problems. Those two problems are
always the same: technical and cultural.

The technical problems I will get back to later.
There are a number of cultural assumptions built into WADL, and those
cultural assumptions come from the WS-* arena. The first is that
you want, and need, machine generated code. The idea that you could get
anything useful from machine generated code is, to me, interesting. If I describe
an Atom Syndication Feed in WADL, how close will the generated code
be to a feed reader like Bloglines? Or even to a library like Abdera? If I write a really good
WADL for (X)HTML how close will the generated code be to a web browser?
The point is that generated code stubs are so far from a completed consumer
of a web service that the utility seems questionable.

The urge to code generate is particularly strong around XML Schema.
You might believe you can talk people out of using WADL that way, but I doubt it; people will
see XML Schema and will automatically look
for the code generation button. That way of thinking is baked into the culture.

The second cultural hurdle is the implicit assumption that all
useful data will be in the form of XML, which ignores
the vast amount of non-XML content on the web today, which, outside
of RSS and Atom, is all of it (HTML, CSS, JavaScript, podcasts, videos, JSON, etc.).
Do I really need to bring up the painful memories of binary content and SOAP?

Q: Ok, so WADL is doomed to repeat all the mistakes of WS-*, only this time with more
verbs, and that failure will be based on completely soft issues like 'culture'.
That's boring. Can you go into the technical details now?

That's an OpenSearch document, that also describes a search interface.

Q: What's the difference?

A: A mime-type.

Q: That doesn't seem like much, does it make a difference?

A: Yes, it makes a big difference.
When you get an OpenSearch document there is a whole data model and a set of
interactions you know are possible because you read the OpenSearch specification.
By reading that spec you know how to construct search queries. When I get a WADL
document it might describe anything, from how to construct a search, to the APP, to JEP, to XML-RPC.

Also, if you are writing an OpenSeach client, you will write something that accepts any
OpenSearch document. That is, the server that provides that OpenSearch document can change
it at any time and all the consumers of that service will 'keep up'.
If it was deployed as a WADL file, there is a strong chance that clients
would be compiled against that WADL, and those clients would
break every time the WADL changed.

So when I say the difference is a 'mime-type', what I mean is that there is an entire
spec somewhere which describes what that document means, and that meaning may include
hypertext functionality, ala (X)HTML, XForms, and OpenSearch.

Q: You say people will compile against a WADL and the clients will break when the WADL changes. How do you know that?

A: You're right, I don't know that for a fact, but I do know history,
and that WSDL was all about the conversion into code. I also know that WADL is so
wide open that you can describe almost anything with it. For example, there is that
WADL for a search interface which I just showed you, and in the same specification there is also
a WADL the describes the Atom Publishing Protocol. How many clients are going to be written
to dynamically consume both of those files? To handle a server switching from one to the other
to describe it's interface?

Q: So there's an example of APP being described by WADL in the WADL spec, aren't they equivalent?

A: First, there are some problems with that example. The first problem
is that it assumes Atom can be fully described by XML Schema. It can't.
The second is the inclusion of the 'app:member-type' element, which is meaningless
in WADL, for which you have to go read the APP spec, which, if you are persistent enough
and actually read this whole article, you will see that makes my point.

But to answer your question, no, they are not equivalent since
APP already has a whole data and service story, and the key point is that that knowledge is triggered by a mime-type.
In reality it's a few mime-types: application/atom+xml, application/atomcat+xml, and application/atomsvc+xml.
Regardless, those mime-types, defined in RFC 4287 and RFC (whatever APP will be), mean that those messages are 'self-descriptive'.
Not 'self-descriptive' in a mind-reading sort of way, but in a 'follows RFC 2616' and 'you can look into RFC 4287 for more details' kind of way.
It is 'self-descriptive' in the same way that HTML resources are 'self-descriptive'; they contain links to other resources,
and in the case of forms, contain a description of a "next-state", either via POST or GET.
In order to use HTML, CSS, and JavaScript you need to read those specs and implement a consumer for them,
which in the case of HTML is non-trivial, and doesn't necessarily mean a browser - think search crawlers.

Q: So every new 'type' of data gets its own MIME sub-type? application/bugzilla-bug+xml and application/bugzilla-bug+json, and ...

A: If that data doesn't fit into an already existing scheme, then yes, it should
get its own mime-type. Think, for example, of image types. We don't have a generic
"image/bin" media-type and force intermediaries to peek inside the binary blob to figure out if you have a JPEG or a PNG.
If something fits into the general form of a collection, then you should be using Atom.
If something fits the general form of a search then use OpenSearch.
If something fits into the general form of a form then use (X)HTML.

Note that APP just describes a general collection, and you don't have to use the whole collection idiom.
For example, a full APP implementation is (list members, create member, get member, update member, retrieve member, delete member),
but if you restict to just (list members) then you have syndication, which is useful in its own right.
If you restrict yourself to (list members, create member), you have a read-only drop box. A collection that just
supports (create member) is a blind drop box. Support only (list members, retrieve member, delete member) and you
have a reviewing queue. You get the idea.

Q: So no one should be designing their own protocols?

A: I didn't say that, just that you should first look out over the protocols
that are available and choose one if it is already close. If none match, then go ahead
and design your own, document it well, and make sure you take the additional steps of registering mime-types
for any new formats.

Q: Even if I just want to define a protocol for my server to talk to my client, which is just a web browser?

A: That's kind of out of scope. We're talking about things that scale way beyond that.
If you are talking about a situation where you control both ends of the wire
and no one else is expected to use that interface, then you don't need WADL do you?

Q: But APP has a Service Document, and if APP needs a Service Document, doesn't that imply that other RESTful services need something similar?

A: Other RESTful services, like OpenSearch, may need their own service description doc, but they should have their own separate mime-type, so consumers know what they "mean".

Q: I don't get that, why not define a standard way to express the introspection information?

A: First, not every RESTful service will require it's own service doc, take HTML for example.
Second, if you define a 'standard' file format for such descriptions, what does that get you?
You give no hint to intermediaries what that file is for, which is the same problem SOAP has,
every request is POST, and every request has the media-type 'application/xml'. There's no meat for intermediaries to dig into
to be able to help. For a concrete example look at the Atom Syndication Format. RFC 4287 is out there, which defines the media-type 'application/atom+xml' and part
of that definition is how to do encryption and signing of Atom documents. That means
it's now possible to build an Atom encrypting/decrypting and signing/checking intermediary.
You can't do that for generic 'application/xml' messages.

Q: People are going to want to describe interaces, how can that be bad?

A: Yes, people want to describe interfaces, and those descriptions are brittle.
If I download a WADL and compile my client today it will break tomorrow when you change
the service. If, instead, you use hypertext, link
following and request construction based on the hypertext and client state, then the
interface won't break when you change servers, or URI structures.

Q: Atom and APP are described interfaces, does that mean that they're brittle?

A: The description of APP and Atom are in RFCs and will never change.
Now, those RFCs might get deprecated with newer versions, but the old versions will never go away, and they will never change.
WADL allows someone to describe a new service every day of the week, and later that week
change the interface.

A: Yes, it seems odd when you put it that way
though I would point out that "flexible == a minimum base of interop carved in stone" with
extensions built on top of that base.

Q: What about the Flickr API? Wouldn't that be better with a WADL?

A: First, you can't expect me to believe that if you had a carefully crafted
WADL you could hand it to WADL2Java and out would pop
flickrfs. Second, if that
had been built on top of the APP then instead of trying to use a WADL generated
client I could start with a generic APP client and build from there, and an APP library
is going to get you a lot further, be a lot better tested, and better adapted to the
task at hand than some generated code.

Q: You don't expect everything to be built with APP, do you?

A: No, there are applications like search for which OpenSearch is a better fit.
And I'm sure there are other areas that would require their own protocols.
I'm also aware of my own blindspots, so why not take a look
at this list and see the kinds of applications that have been
given APP interfaces.

Q: So we've talked over the pragmatics, but how does this come from REST?

A: Much of this is driven by two constraints. The first is obviously
"hypertext as the engine of application state", but another one even less frequently cited
is "a constrained set of media-types". Think of the two extremes. If there was only one
media type, 'thing/thing', then you wouldn't be able to do anything useful. The other
extreme is just as bad, if there was a huge amount of media-types floating around the web
then you could never use it either. Think of trying to build a browser and having an endless
list of image media types to implement. This also affects intermediaries, think of those
accelerating intermediaries some dial-up provders use to reformat images on the fly so they
are smaller. Those would be impossible to write unless there were a constrained set of
image media types. Given the history of SOAP and WSDL, I believe WADL will be used to create a whole
slew of resources with a single media type. Even worse, when you look inside them, they will
probably represent a nearly infinite set of new document types.
So the idea is to re-use existing protocols like OpenSearch and APP, that keeps the mime-type
manageable.

Nicely put. This clearly makes the point that self-descriptive data is more important than IDL.
The treatment of brittle/flexible/carved in stone didn't quite work for me.
From Robert Martin's "Design Principles and Design Patterns"[1] there is a definition of Stability that I think could apply here as well.
"Stability is related to the amount of work required to make a change."
In that paper a software module is "stable" not because of anything inherent to itself (high quality, well proven design, ...), but simply because other things depend on it.
Depending on APP is a safe thing to do because the standard is going to be very stable (because many servers and clients are going to implement and support it).
Also, APP is designed with flexibility in mind; clear extension points and semantics for how to handle unknown data.
[1] http://www.objectmentor.com/resources/articles/Principles_and_Patterns.pdf

Hi Joe, excellent write-up. I wrote about the half-baked idea of a dynamic description language a while back. One potentially capable of evolving over time. I never really saw the idea through though. I get where your coming from here, the brittleness of WADL, why I'd never want to use it, but don't see that a WDL has to be that way.
If depreciation and upgrade paths can be defined in protocol, providing clients with time to adapt, a persistent URI, least-amount-of-validation(preferably none) and machine/human readable documentation, then I think a protocol that doesn't require registering a new mime-type whenever an interface changes could aid in innovation and adoption.
Personally I don't believe seperating the protocol spec from the resource it describes is a good idea. It should all be in one. The entire resource, interface and document should be self-describing. Not require that a client go to another resource that could itself break(Standard disappears), for information on how to interact with it.

> if you had a carefully crafted WADL you could hand it to WADL2Java
Well, for a lot of today's APIs (Digg, Yahoo!, Del.icio.us, Amazon E-Commerce, ...) REST Describe & Compile does at least a not too bad job. It is WADL2Anything, kind of :-) At present it supports Java, PHP 5, Python, and Ruby. However, the application is still in development and far from a 1.0 version.
Try it online: http://tomayac.de/rest-describe/latest/RestDescribe.html
Source code: http://rest-api-code-gen.googlecode.com/files/rest_describe_v0.3.zip
Details: http://blog.tomayac.de/ViewByCategories.php?category=Work
Cheers,
Tom

If […], then I think a protocol that doesn’t require registering a new mime-type whenever an interface changes could aid in innovation and adoption.

Did you read the last paragraph of Joe’s post? That was precisely his point: there should be neither a new MIME type for every single service nor a single MIME type that all services use. Your proposal seems to come so perilously close to the latter that I wonder whether you understood Joe’s point.

Personally I don’t believe seperating the protocol spec from the resource it describes is a good idea. It should all be in one. The entire resource, interface and document should be self-describing.

Can you give any reasons for this? Clients are not going to be able to infer meaning anytime soon (not before AI exists), so what you’re saying amounts to saying that no code should be reusable between services (or even between resources of a single service).

We have proof beyond any doubt that centrally defined, sufficiently specified and widely implemented standards work and that they’re a good idea. (Cf. IP, TCP, HTTP, XML, etc.) What sort of advantage do see in the model of “one media type per resource” that could outweigh both the approach’s own drawbacks and the loss of the benefits of “the internet way” we’ve had so far?

Not require that a client go to another resource that could itself break(Standard disappears), for information on how to interact with it.

You have to be… really paranoid to deploy a service on the internet and then want to protect against the eventuality that an IETF standard would disappear…

Nicely put. This clearly makes the point that self-descriptive data is more important than IDL.

Thanks!

I would have to disagree about stability though. Atom, and soon the APP, are fixed because they are carved in stone. Even if no one implemented them, those RFCs would remain unchanged. You can't believe how important that is.

Also, APP is designed with flexibility in mind; clear extension points and semantics for how to handle unknown data.

I've put up a new blog entry declaring that REST relies on data contracts and not programming contracts.

Let's change the conversation from "REST doesn't need an IDL" to the more positive "REST already has contracts, and here's how...".
The discussion above is part of that, but I also really liked Alan Dean's Shopping RDF example:

To validate that the server supports the "Shop" protocol, the User Agent can test the document for the existence of the basket element, e.g. with the following XPath expression count(//shop:Basket)!=0

If I download a WADL and compile my client today it will break tomorrow when you change the service. If, instead, you use hypertext, link following and request construction based on the hypertext and client state, then the interface won't break when you change servers, or URI structures.

If there any evidence for backing that claim up?

I'd like to know of some actual examples where people did the "right thing" (which, here means at least that they did *not* explicitly describe an interface using anything like WSDL, WADL, or, presumably, plain text), then changed servers or URI structures (though, honestly, 'URI structure' is a bit vague) *and* the "interface didn't break".

I'm not entirely sure what "interface didn't break" means, here, but I assume it means, at least, that no client code had to change in any way.

(FWIW, I think there's *value* in clients breaking when interfaces break, so long as they break loudly and obviously. I guess I'm in the Torvalds School of API evolution in that respect. But that point is orthogonal to your claim, which I find quite curious.)

Yes, this is one of those things that works so well that you will be prone to claim the examples I'm about to give are "too obvious" or "too simple".

The first one is the most obvious, and you just used it. I've changed my comment form many times over the years, yet everyone is still able to use it. Even the spammers. That was until I came up with my latest markup-based captcha technique, which is so far 100% effective.

I've also moved my syndication feed around a couple times, and I could easily re-arrange my URI structure of my blog and people subscribed to my feed would not notice any difference, even when clicking through to an article.
(Yes, syndication is a web service, one of the earliest, simplest, and most powerful.)

In the process of writing my APP server implementation for this blog I changed the URI structure a few times. I never needed to update the client.
(But this goes to show how ingrained that kind of breakage is in my psyche, I panicked one time when I launched the client right after updating the server, convinced that I forgot to 'fix' the client to work with the updated server. It was a pleasant, "aha, of course it still works" moment when it come up fine.)

Also, very early versions of my APP server code handled only 'content' and not 'summary'. No client code needed to change when I added 'summary' support.

I'm pretty sure you could find similar stories for OpenSearch, but I don't know any first hand.