As with all but one xkcd April 1st plan[1], this was conceived briefly before the 1st and rushed into production for the 1st. One year we will plan ahead but that year is far off. This project had, for one of our April 1st events, an extremely long timeline in fact of several days. Usually less than 24 hours is available for implementation.

A short reflection on what went right and wrong seems warranted.

Things that went well:

Initially Waldo.BrowserCap.mergeBE was a mess of fairly imperative code. Due to many errors in the dataset it was iteratively cleaned up until it became the current functional implementation. This implementation is far more resilient, shorter, and clearer. While this repository does not show the progression it was a beautiful progression of iterative refinement from imperative to functional styles that reminded me the benefits of the lazy programming.

Testing via Waldo.LoadTest. Early on we launched a small script that recorded the requests waldo would receive. Our testing merely was to replay these, as they’d come in through the heart of waldo. While not exactly matching release day requests they were very similar - on release day people poked at the code base and more unusual requests were made - allowing testing of both stability and performance. A typical test run involved several million requests. On April 1st we used the actual April 1st traffic to test improvements (primarily for efficiency). This testing was both cheap and effective catching every bug we are aware of in the core code prior to release. The only bug we know of that actually made release was a bug in the stats system we used to see how fast people were receiving each possible combination of each story. This bug didn’t show up until April 2nd when we’d gone to bed and stopped checking the stats religiously. Why? Because that bug was an issue of us building up too many function applications. If they weren’t forced quickly enough it caused a stack overflow. Notably this was not tested by the LoadTest module but it was explored with said after it was discovered. As the stats system was a quick hack that had been added around the start of the 1st and was no longer needed for our use, the decision was merely to scrap it instead of fixing the actual problem. A more user friendly stats system is wanted for anyone else using this code base in the future anyway. In fact that was the only space leak encountered and would have been trivially fixed had fixing it been warranted.

Using an embedded domain specific language to encode the stories. The stories came together at the very end and the eDSL made encoding them fairly fast and easy. Even an experienced occasional programmer was able to quickly make changes and additions. I believe that without this level of abstraction that would not have been possible and I do not believe that directly encoding it would have actually saved time. The eDSL is still far from optimal though. The Waldo.A1 module, which implements the Umwelt story script, still comes out to 969 lines of code.

Performance of the Haskell code was very good in both terms of throughput and latency. No extensive tuning was required.

chromakode doing the front end side. He’s amazing and reddit should give him back.

This that went wrong:

There are very few comments - even where they would be fairly important. Time for typing was time lost in this project sadly. Not only comments suffered for this.

Waldo.BrowserCap uses a rather poor matcher. This requires that we try every possible browser separately and with crafted inputs some matchers can be sent down the garden path. For all the complexity of the API of the Haskell regex libraries, I was unable to use it ways I would have used regexps in other languages. Even this style matcher, encoding all of the browsers as a tree would be an improvement though (still suffering the efficiency problem though). Sadly, the timeline of this project did not allow much time for optimization and as such the simplest solution was used and a large LRU cache was placed in front of it.

Possibly the eDSL should select the panel sizes also. A knapsack sizer may be a poor choice even in the current form and is pluggable. Time disallowed playing with other solutions.

Future plans:

Split out Waldo.BrowserCap as it’s own module. This really isn’t something that is conceptually part of waldo and it would make sense to use it elsewhere.

Instead of integrating the server, export an API that can be used as a component. This is how it was really designed to operate but this was more convenient at the time. The benefits of rolling it into dynamic.xkcd.com like I eventually plan would have actually detracted while we were so actively working on it.

Many parts of xkcd are implemented with Haskell. A short list includes:

dynamic.xkcd.com (though it is unloved and in need of a rewrite, using an old, old version of happstack - a rewrite on WAI exists and will be deployed soonish)

Umwelt, which will be merged with dynamic.xkcd.com soon; described above.

pbfcomics.com which we host and somewhat maintain. Based on WAI, feed, and HStringTemplate (mistake due to the Haskell implementation of StringTemplate).

holistic.xkcd.com a quick joke site using WAI, digestive-functors (plus internal digestive-functors-wai that needs to be open sourced), the leveldb bindings, and for spam checking http-conduit.

Two new sites that haven’t been released yet. One of which is written using Hakyll.

Haskell is used so much over here because of its safety and low surprise factor. The community is made up primarily of contributors who tend to care about the details of their respective problem domain and craft APIs to match. While libraries can be scarce the ones that are available are crafted with an attention to detail often lacking in other languages.

We have a decent amount of code, but we’re not actively maintaining all the pieces all of the time. When we aren’t working on our code, it needs to keep on working, and when we come back to it after a long absence we need to not have to remember that something in a far flung file interacts with it - there must be a clear way to determine the exact behavior. Pure functional programming provides us with very strong guarantees about isolation between the different parts of our code base, while strong types give us assurances about behavior and let us lean on the assumption that in Haskell, “if it compiles, it is probably (still) correct.”

[1] The comic swap was conceived as a swap of news sites initially about a month ahead but shortly before the 1st was changed to be comic sites due to politics.

I doubt this will save perfromance, but it's easier to read, I think. comparing is from Data.Ord. Nice function. I'm not doing some code review, by the way, just perusing! Code that's been hit by millions of users is always interesting to look at.

While an easy problem to hit it is also usually easy to fix. This is the first time it made it past my basic sanity check line and out "into the wild" of actual use and not just something I bump into during development.

Probably due to my reliance on the stream-replay testing which did effectively protect the user-visible portion of the app.

I think the worst part is I had discounted that happening because I used a strict data structure - of course it never actually made it INTO the data structure ... I'm blaming 0200 on April 1st for that one.

I'm not really prepared to provide an educated discussion on it ATM. I rewrote that site back in September. If a proper discussion is warrented you probably should email me and maybe I can try to dig up the specific details.

From my fallible memory though:

Performance, specifically latency was an issue and we stick a CDN in front of it to alleviate that. This specifically is why we abandoned plans to take a similar approach with xkcd.com it's self.

Some important variations from the StringTemplate spec. that interfered with our usage.

The implementation didn't make me want to approach the problem by improving it.

Please do shoot me an email about it. HStringTemplate was one of my first real Haskell projects and I'll willingly cop to some flaws in it (particularly the dumbly super-pointfree code which makes it unpleasant to work with). I thought I stuck close to the 3.1 spec (there was no 4 spec when I wrote it), with the exception of newline handling which irritated me in the 3 spec so I did more straightforwardly.

As far as performance, did you try using the Text or ByteString backends? Performance is one of the arenas where' I've heard few complaints in the past, so if you have some testcases which I could profile, then that would be helpful.

I'm a big fan of pbf, so it was a really pleasant surprise to hear that my code was being used to serve it up, and simultaneously a downer to be told that there were problems with it -- so I'm happy to see if there's anything I can do to help :-)

Also, it might be possible to add just some better template call syntax without implementing the whole ST4 spec, if that would help.

I'm using Text. I've very rarely used Strings in Haskell code actually. Infact it looks like I abstracted away most of what type it was using at some point - perhaps I was playing with strict vs. lazy Text.

As for performance, we might have very different definitions there. HStringTemplate works fine for pbfcomics.com but xkcd.com is another matter. I'll throw it back into the profiler to re-familiarize myself with that part. A quick ab run though suggests the code base only supports around 75 qps with a latency of over 10ms per request. Of course I'll need to spend a little time digging to be sure how much of that was HStringTemplate - with numbers like that I had to take a very different approach to performance then just improving the server.

Yeah. For xkcd's numbers, I'd probably not want to do any template rendering on the fly. I might imagine that the latency issues in hstringtemplate itself would have to do with nested template calls and similar logic more than just plugging in a few holes? If so, I imagine moving to more modern data structures could be a real win there. But even then, the template engine is an interpreter and not a compiler (although a stringtemplate compiler would be pretty sweet), and so there's some necessary and unavoidable overhead.

w/r/t the syntax, if you have any concrete proposals to make (for template calls or otherwise) please let me know, and I'll be happy to look into it.

Also, it implements an old version of the spec which is less pleasant to work with. Not decisive in it's self but it helped drive me off. (This is the sort of template system I've always preferred and gone out of my way to select.)

For example, "To include another template, just reference that template like a function call with any arguments you need. Note that expr is either a template name or a parenthesized expression that evaluates to the name of a template. E.g., (templateName)()." - ST4 spec

That last part is very beneficial. With HStringTemplate you end up having to write a long "if" chain where ever you include a large number of templates IIRC. This can add a fairly high overhead to using it.