Tuesday, September 23, 2008

Webmachine is a resource server for the Web.

I don't like web frameworks. They're mostly-complete Web applications with a few places you get to modify. If your biggest goal is to get something running instantly, and your something is shaped enough like an existing framework, then it might be your most effective way to get started. However, as soon as things inevitably get interesting, or you want to use a a different data storage system or some aspect of HTTP that your framework didn't envision... well, your handy framework is now something you have to spend time working around instead of with.

One of the first technical questions to deal with in a Web startup is "which Web framework should we use?" If the term wasn't already taken I would have imagined a Web framework to be something that gave you a place to define the representations and other Web-relevant properties of your application's resources -- with the emphasis that the first-class things on the Web are resources and that their essential properties of interaction are already quite well defined and usefully constrained. It was a Web framework about the Web that I really wanted, but I didn't see any.

I knew I wasn't the first one to ask. People like Bill have been wondering “where are the REST toolkits?” for a few years, but nothing really strong has yet emerged. Mark Nottingham has pointed out that even the better Web frameworks don't realize the power of having generic semantics. I agree, but I think that his examples didn't quite go far enough. Last December, Andy Gross and I realized that we could write something good enough for our own needs easily -- so we did.

Introducing Webmachine: a layer that adds HTTP semantic awareness on top of the excellent bit-pushing and HTTP syntax-management provided by mochiweb. Webmachine is not yet complete, and it's not perfect, but it has worked well for us and we'd like for there to be more conversation about what REST tooling should be like.

The initial implementation of Webmachine was inspired in large part by version 1 of Alan Dean's HTTP flow diagram. I subsequently worked with Alan to improve the diagram as we developed Webmachine. Webmachine's request handling engine reflects that diagram explicitly.

A brief language digression: when we started writing a system in Erlang we assumed that we'd only be able to use it for some special-purpose backend work. We were wrong. It's been great for rapid development of interesting Web applications, and developers seem to like it. All of our server-side work, from web request handling to page templating to object storage has ended up being in Erlang not out of any dogma but because it has been such a practical tool.

At some point I hope to also post here about our choices in templating language, our unusual decisions for data storage, building practical systems around linked data, and how we've gained business value from representing our core data in multiple media types. But part of the point I hope to make with Webmachine today is that those choices ought to be separable from your choice of Web / REST tools.

10 comments:

I just saw that you posted webmachine to google code, I was actually in the process of building out a similar framework which we were calling Dissident http://github.com/voodootikigod/dissident/ I had not gotten too far on the backend, which from what I can tell was going to be similar if not exact to webmachine. I was wondering if you wouldn’t mind us using webmachine as the backend and putting our AJAX head design concept on top of webmachine?

This is pretty awesome. This is pretty much the effect that I was trying to achieve with my latest web "framework" (although it's really a library) mulib:

https://wiki.secondlife.com/wiki/Mulib/Documentation

The design philosophy is to stick to helping the programmer implement REST Resources and then just get out of the way. It doesn't have any templating or database access abstractions, so you can just plug in another one or write your own, if you even think you need those things.

I'm definitely going to be checking out webmachine and stealing any good ideas! :-)

Thanks very much for publishing this - like some of the other commenters, I've also written a dispatching system against mochiweb but prefer yours quite a bit.

One question I have is about language negotiation support - I notice that you have content type negotiation but couldn't find the equivalent code for languages. Does this exist? If not, do you have short-ish term plans to add it or pointers on where in the code I could add it?

With regard to negotiation, I'm enabling it one type at a time. The initial release supports basic accept (content-type) negotiation. The next release, in a week or so, will have full accept-encoding negotiation and the subsequent one will have accept-charset negotiation. I also intend to add accept-language negotiation soon and would be happy to discuss it with you ahead of then if you wish.

The mechanism is easy to plug in; the tricky part is just that each type of negotiation has slightly different details via RFC2616 and so (while they do share some code) they must be written one at a time. If you're interested in writing the accept-language negotiation in order to get it done even faster than I will, just send me email and I'll point you at the right starting places.

What's the best way to submit fixes for webmachine? The google code page lists you as an owner but does not reveal any email address.

The one little bug I just ran into regards decoding of the etag request header. In webmachine_decision_core:decision/1 (v3g11 and v3k13) there are calls to mochiweb_util:unquote_header, which should be changed to mochiweb_util:unquote since only the function unquote is exported by mochiweb_util.

I use my google account (justin.sheehy@gmail.com) for maintaining both the google code project and this blog.

And right you are -- we've had a version of mochiweb running with that exported for so long that I forgot it wasn't that way in the standard mochiweb distribution. I will fix that in webmachine-0.19. Thanks for bringing it to my attention.