Archive for the ‘Scheme’ Category

This is the story of how I deployed the Taubatron – a web application written in Scheme. I wrote about what it does earlier; this post is concerned with getting it running on the actual internet, as opposed to on my personal computer.

CGI vs server process

The most straightforward approach would have been to deploy the application as a CGI program. I had deployed Scheme applications at my hosting provider that way before. This is also the type of service offered, at the time of this writing, at the free Scheme web hosting site ellyps.net. But for this application, performance with CGI was a problem –

The slowness – up to several seconds to complete a request – had two causes: first, the expense of forking a new guile process for every request, and second, the application’s lengthy initialization phase (building a graph out of a dictionary of words). Web developers in other languages have found ways to avoid these costs – for Python there is WSGI, and of course there is mod_perl – couldn’t I do as well in Scheme? I considered mod_lisp and FastCGI but frankly these seemed difficult and perhaps not possible on my host. The approach that seemed most promising was to run the application as a long-living server process using the built-in HTTP server found in recent versions of the guile Scheme compiler. Getting such a server application running was about as easy as setting up the CGI program and the performance boost was remarkable:

That’s right – performing the initialization steps just once resulted in a 100-fold performance increase. Serving requests directly from the built-in HTTP server like this probably represents a lower bound on the latency of this application. But to use this approach at all, first I would have to get a recent-enough version of guile running on my host, which turned out to be non-trivial.

Installing guile on the host

Of course guile did not come pre-installed on my host the way PHP did and I had no root access so I could not simply install it like I would on a personal machine; however I did have user-level shell access and the necessary tools to compile guile from source. This is how I had gotten an older version of guile running to host some simpler applications. But the high-performance web server is found only in a newer guile version which failed to compile from source due to resource limits imposed by the hosting service. I tried simply uploading a build of guile from a Debian package appropriate for the hosts’ architecture; this failed at run time with an error about a glibc version mismatch. However, I noticed that the early parts of the build process that involved compiling and linking C code were working fine; the build wouldn’t fail until later when guile was trying to compile parts of itself into bytecode (files with extension ‘.go’ in the build tree). Figuring that these bytecode files might be architecture-dependent but should not depend on any specific glibc version, I tried simply copying to the build tree from the Debian package those bytecode files which were failing to build. And it worked – I had a working guile-2.0 compiler installed on my host.

Configuring the front-end server

But of course I wasn’t finished – it’s not as though I could just bind the guile server to port 80 on the shared host and be done with it. I needed a way to integrate it with the front-end server, Apache in the case of this host. One way is to bind the guile process to some high-numbered port and use Apache’s RewriteRule to map requests to that port. But in a shared hosting environment I couldn’t count on just being able to grab an available port. I had a good discussion about this at Barcamp Portland and concluded that the best approach was to bind the guile process to a local unix socket, and then configure the front-end web server to forward requests to that socket. Binding the guile-based HTTP server to a unix socket was no problem, but trying to figure out how to get Apache to cooperate in this seemingly simple task was frustrating. I eventually tried asking the Internet, but apparently it either did not know or did not care. In contrast, it is easy to findexamplesof this in nginx. I soon had my application serving requests through a unix socket behind nginx with a latency of less than 3 msec per request – nearly as fast as the bare guile HTTP server. (It entertains me that this benchmark, my first use of nginx for any purpose, was achieved on a laptop on Portland’s MAX blue line, heading East.)

The CGI trampoline

Still, I was not finished, because I didn’t have the option of using nginx on my host – I had to figure out a way to make their Apache installation work for me. I gave up on finding an Apache configuration directive to do this and realized that there was a alternative that was also likely to be portable to just about any host, no matter which web server it was running or how it was configured – I could write a lightweight CGI process that would simply open up a connection to the socket, forward the request, and echo back the response. I called this a “CGI trampoline”, implemented it, and after the fact found at least one other implementation of the same idea using the same name. My first implementation was in Scheme, and I had my web application serving requests through a unix socket behind Apache with a latency of 39 msec – ten times slower than the bare guile HTTP server, but still ten times better than the whole application as a CGI process. The performance hit was due of course to the cost of starting a new guile process for each request. I rewrote the CGI trampoline in C and request latency dropped to 3.6 msec – pretty good compared to the lower bound of 2.4 msec achieved by serving requests directly from the application running as an HTTP server.

When I needed to develop web applications for the first time, naturally I turned to Scheme, a language which has been central to my evolution as a programmer and which remains my go-to language for many things. I am still an enthusiast for web development in Scheme, and I think it could become much more popular than it is, particularly if it were easier to find hosting and support.

All the libraries you need for Scheme web development already exist. One of the earliest was guile-www, which provides support for HTTP, cookies, CGI, and related things. For this post I turned to the current maintainer of guile-www to ask what motivated him to become involved in Scheme web development. It appears that, perhaps because Scheme remains something of a fringe language on the web, the ecosystem around it continues to evolve rapidly. Projects fork, merge, and fork again; there isn’t any one “Scheme stack” that dominates. Nevertheless, the basic reasons for using Scheme for web development – it’s simple, it’s concise, it’s elegant – remain the same.

What drew you to get involved in maintaining guile-www?

ttn: Back in 1998 I made various motorcycle trips, and published a “trip log” with photos to a website. This used some custom Emacs Lisp to push the html and jpeg files to the server (you might find wup.el — web update — somewhere on the net). I had just discovered Guile and was intrigued by its module system, certainly a step up from Emacs’ crowded flatland.

I stumbled onto Guile-WWW, which IIRC at that time was client-side and HTTP/1.0 only, and started fiddling with it. Around 2000 or 2001, I somehow also finangled write privs to Guile proper (the umbrella project that also included the CVS sub-repo of Guile-WWW) and started landing patches there. Unfortunately, due to a misunderstanding with the former Guile maintainer (Marius Vollmer), those privs got cancelled.

Since the code is GPL, i started maintaining the package privately, naming myself as the copyright holder for new code and making releases every so often. The hope was (and continues to be, though ever more faintly, seeing how things have developed since) that after some time, my changes could be merged back to GNU Guile.

What advantages do you see Scheme having over other languages for web development?

ttn: Less typing and more fun: My Emacs inserts () instead of [] automatically, edits and motions are by sexps, Emacs boldfaces the current list, etc.

Do you have a favorite application that uses guile-www?

ttn: That would be Sizzweb , i suppose. I use it to serve Debian ISOs and other stuff on the home network. Client-side, not so much happening…

The development of guile has picked up in pace in the past few years; do you see its niche growing and that of guile-www with it?

ttn: Yes. I believe bundling (web …) modules in Guile proper was a strategic mistake that renders Guile-WWW independence even more important. So “with it” might be better read as “parallel to it”.

So web developers do use Scheme, even if it is far from the most popular web language. But web development in Lisp is not just possible: it is also a good idea, especially as your project evolves past “a page with some dynamic content” to “a full application that runs over the web”. HTML can be understood in two ways:

As plain text with a few formatting tags thrown in

As a tree of nodes, where nodes have names and optionally, attributes and values

These two views lead to two different approaches to dynamic web programming:

Treat the web page as a plain text document, with little embedded snippets of code that expand like macros when the page is displayed

Represent the entire web page as a tree using a native data structure, which is then rendered as HTML at page display time

Both HTML and many web programming languages were designed with view (1) in mind because it can be very simple. You don’t need to be a programmer, or to understand what the phrase “tree of nodes” means, in order to understand what this is supposed to do:

Today is: <?php echo date('Y-m-d'); ?> and it is a <i>good</i> day

But the problem with this approach is that it doesn’t scale well to more complicated uses. Take, for instance, this production PHP snippet from the WordPress code base (pasted here as an image because of the difficulty of embedding actual literal PHP here):

That, despite being an example of well written PHP, is not terribly intuitive even to a programmer. I really dislike the intermingling of two distinct syntaxes (that of HTML and PHP). It forces you to switch back and forth between the two languages when mentally parsing such code. It leads to odd orderings of delimiters – <?php { ?> etc. It’s what leads to PHP being called things like “bloated, messy, and counter-intuitive“.

Such contortions do not appear in web applications that treat the whole page as a native data structure which the system then renders as HTML. Since all of the actual HTML is generated by the renderer, the application developer need not even know any HTML syntax in order to program this way! Scheme is well-suited to representing such a tree of nodes as a native data structure, and there is a standard convention for doing so known as SXML. One of the things I like best about Scheme for web development is the clear equivalence of a live Scheme object, its serialized representation as SXML, and its serialized representation as HTML – all of which can be interconverted with a single command.

Interconversion of HTML and its equivalent Scheme representation

This means we can take the result of any Scheme computation, manipulate it into SXML, and render it as HTML. A working implementation of the following example can be found here. Consider a function top-word-usage which splits a string into words and returns the number of times each word appears, sorted with the most common words first:

By now JSON is a widely-accepted standard, well-supported in the most common web development languages. But the whole point of an open standard is that one ought to be able work with it in any language and on any platform. Attempting to work with a standard on an unusual platform is a good way to test how “open” it really is, and standards have good reason to pay attention to the results: not only do fringe players sometimes become more important (see, e.g. Linux kernel), they also help keep standards honest by preventing language-dependent or platform-dependent “conveniences” from sneaking in.

So when my 13-year-old brother-in-law Lucien and I arrived at APIhackday, we decided to work with web API’s in two of our favorite languages – lua and Scheme – even though these are not often thought of as web programming languages. We chose to work with PDXCouncilConnect, which provides access to civic records of the City of Portland through a web API. A typical JSON result stream looks like this:

A simple JSON parser can be written quickly, but it is also possible to find at least a rudimentary one in almost any language. Lucien found a lua JSON parser, and using the CouncilConnect stream, actually found a bug, wrote a workaround, and by the end of the session was parsing the JSON stream. Though there seems to be no dominant JSON parser for Scheme, I quickly found a simple one. Now that I had the JSON stream as a Scheme object, what to do with it? My first idea was to pull in the guile-gnome library and use it to build a graphical desktop app (using GTK+ widgets) that draws its data from the web stream. By the end of the one-day hackathon it was working, demonstrating the utility of Scheme in pulling together disparate libraries quickly. Though interesting, this result didn’t seem to harness the unique advantages of the Scheme language. After all, JSON comprises nested lists of values, and manipulating lists is what Scheme is for. Couldn’t we use Scheme to transform the data in some useful way? Here is the CouncilConnect stream, represented as a tree:

JSON input, represented as a tree

As I watched people around the room working with this API for the first time, I noticed that I and others were spending much of our time studying the JSON stream to understand this underlying tree structure. Though we often knew immediately which element we were interested in (for example, the “votes” node, highlighted), we first had to learn how to navigate the tree in order to get to it. What if there was a way to bring this part of the API, buried deep in the tree, to the forefront without having to understand the whole API? We can re-root a tree at any node by changing the direction of some of the links (visually, grab the new root and hold it up; let the rest of the tree dangle down):

.

JSON tree, re-rooted at "votes" element. Edges that have been flipped are highlighted

Now, the most interesting piece of information is easily accessible, right at the top of the tree. It is generally easier – both conceptually and in code – to navigate down a tree than it is to backtrack up it: in the rotated tree, I can still see that the “votes” object is part of a “motions” object which is part of an “items” object which is part of a “session”, and I can still navigate those relationships, but I don’t need to understand them just to get at the “votes” data.

It is possible to write a completely generic Scheme function that takes as arguments any JSON tree and any node-matching predicate, and returns a tree re-rooted at the node matching the predicate (or a list of such trees where more than one node matches). This is the sort of thing Scheme excels at; the heart of the code looks like this:

Now, having the re-rooted tree, what should we do with it? How about de-parsing it straight back into JSON? The Scheme program is now acting as a sort of middleware or API virtualization layer – the re-factored JSON stream can be fed to any other JSON-capable tool (in any language). Here is an excerpt of the CouncilConnect JSON stream with “votes” rotated to the top:

Now, I can easily access the contents of the “vote” object at the head of the stream, and if I care to, I can drill down further to see that the vote was passed and that it was related to a Sewer program.

For another example, let’s hoist the node called “bureaus” to the top and see what happens. In the result below, we find at the head of the JSON stream a piece of business related to the transportation bureau; navigating down a level, we see the result of a vote; down one more level, and we see that this was about disabled parking permits.

So, there it is: an application of Scheme code to web API programming, which leverages the strengths of the language, and which might even be useful. Here is the full code for the JSON-refactoring program: