Wednesday, October 22, 2008

The BeeBole Erlang/Web Tutorial, Webmachine-Style

[EDIT: more up-to-date and very interesting versions of this app rewrite have been done by Bryan Fink (using a Riak backend) and by Serge (updating for Webmachine 1.x and implementing some of the improvements I suggested here). Either of those versions should be considered a better place to start than the code linked from this post.]

Mochiweb is a fantastic programmable HTTP stack - but it's not a Web application toolkit. For that reason, when writing an app using only mochiweb one tends to do exactly what Hughes did in his tutorial: put several logical layers of functionality together in one big loop. You could argue "it's a tutorial on how to use mochiweb", implying that you might well benefit from having an amalgam of functionality (e.g., protocol, application code) exposed. While true, that also obscures important details and muddles the inherent value of separate logical layers that a good Web application embodies.

Webmachine, built on top of mochiweb, allows you to focus on writing your app while ensuring that things like caching behavior, content negotiation, and response codes, to name a few, are both purposeful and correct; think of it as an ambassador to HTTP: a lightweight, higher level framework for utilizing the excellent mechanics of mochiweb in ways that maximize your ability to focus on what you set out to accomplish: making a good Web application.

This all sounds nice, but -- to help me decide if it actually helped beginners in that way -- I tried a quick exercise of implementing Hughes' tutorial application as a Webmachine resource.

(note: The snippets here, concatenated, make up the entirety of the resource code. This really is all there is to it.)

I started by creating a Webmachine quick start directory and putting this dispatch line into the _sup file: {dispatch, [{['*'], stickyNotes_webresource, [DocRoot]}]}]

This causes all requests to be sent to the stickyNotes webresource module, with their full path passed along and the DocRoot used as an initialization parameter to that resource. If I was doing this from scratch I'd have broken up the "notes" data and the static file delivery into separate resources, but the goal here is a simple work-alike. Now we need to write stickyNotes_webresource.erl:

The module and include_lib lines make up the sum of our boilerplate. The export declares which of the Webmachine resource functions we intend to override, and the record declares the data structure we'll use as our context threaded from function to function: a simple two-element struct.

init([DocRoot]) -> {ok, #context{docroot=DocRoot}}.

The Webmachine resource gets initialized for each request, and stows the docroot away for later use in serving up the static parts of the application.

This declares that the URI /notes will only handle POST operations, and all other URIs served will only handle GET and HEAD operations. If a request is made against a URI with a different method from those allowed here, Webmachine will send a 405 Method Not Allowed response code.

That behavior is different from Hughes' tutorial code, which sends a 501 Not Implemented, regardless of URI, where the method is other than GET, HEAD, or POST. In that original tutorial, if a request goes to a working URI but with the wrong method (e.g. GET /notes) then a 404 Not Found is sent instead. Very confusing!

One of the core purposes of Webmachine is to encapsulate as much of the semantics of HTTP as possible so that applications don't have to recreate it. It's very easy for a smart person to get these details wrong, so there is value in not making them do it over and over again.

This is conceptually simple (resource_exists simply decides whether to send a 404 or to proceed) but it ends up being the longest function because I decided to be explicit about checking the correctness of the URI path.

Also, this is where we set the fullpath element of our context object in the cases where we are going to serve a static file.

These functions perform our content type negotiation and deliver the actual responses in the case of static file delivery. The first case entry ("") is there to account for the special index.html at the root.

This, the last part of our resource definition, handles the POST requests. All of these are already known (via allowed_methods) to be to the /notes URI path.

I would have done a few things differently than this if I was creating a tutorial from scratch. For starters, I would have done all of the JSON fetching ("read") in a GET operation instead of tunneling it through POST. I also would have made the POSTs directly in the media type application/json instead of burying the JSON inside a form-encoded body. Minor things like good HTTP behavior aside, it's a nice little demo and engaging enough to be a fun start for those new to Web programming in Erlang.

If this application was rewritten to have notes be proper addressable resources, it would then be trivial in the Webmachine variant to add the optimizations that were (sensibly) left out from the tutorial such as content-encoding selection or etags and other caching support.

Mochiweb is a great and powerful HTTP programming system. Webmachine, atop mochiweb, gives Web developers a higher-level platform to program against. This allows them to more easily develop general-purpose and extensible Web applications. The Webmachine source is currently available via SVN and soon to be made available in versioned tarballs. I've also made the Webmachine version of the tutorial code available in case anyone would like to play with it. If you download the tutorial and you already have a working Erlang/OTP installation, then you should be able to get running simply by:

cd /tmp (or wherever)tar xzf webmachine-stickynotes.tar.gzcd webmachine-stickynotes./start.sh(in the erlang shell, the very first time, run stickydb:reset().)(point a browser at http://localhost:8000/)

This is great stuff. As a small company just getting started with Mochiweb this will really help. I was looking for a super fast REST framework and thought I had found it with Mochiweb but this goes even further! I was wondering if you might give some more insight into the data analysis pieces you have built with Erlang if it's not too much secret sauce. I know you have released some of the components with distributerl but I was looking for insight into how you all answer arbitrary data analysis questions with your homegrown backend.

John, I'm glad to hear that Webmachine might be a good fit for you. There will be another release fairly soon with some bugfixes and improvements. Please feel free to send me email whenever you wish, especially if you're using it to build anything new.

We're not prepared at this time to release any of our data analysis code beyond some basic building blocks like those in distributerl. There will certainly be more along those lines released later -- it will be used commercially and select portions will continue to be released as open source.

Combining webmachine with erlydtl gives me allmost all I need for speedy web development. It's remarkable easy to combine, I just had my first page up served from webmachine showing an earlydtl template in an hour or so; with no experience with both libraries (although knowing the django template syntax ;)

I am really liking webmachine and the wmtrace facility (add by Bryan right?) is so helpful.

Thanks for the stickyNotes conversion. Maybe it's time to update it to webmachine 1.0? Although with the wiki guide it was not hard for me to upgrade it myself ... helps learn the framework too I guess.

The only thing I find missing is the ensurance that mnesia is started. I don't find the equivalent of 'stickyNotes:start()' in the webmachine version. So I either start it or reset it from the erlang shell, but that isn't the preferred way of doing it ... is it?