Bλog

Links

Queueing IO actions

Creating a simple queue system for retryable IO actions
Published on April 18, 2011 under the tag haskell

Lately, I’ve been working a bit on an automated system to count laps for a running contest called 12 Urenloop (Dutch). We’re working on this application with our computer science club, and I’m writing the counting backend in Haskell (the repo for the full application is here).

I came accross a problem that turned out to have a quite easy and elegant solution – or, at least, more easy and elegant than I initially thought.

After cleaning it up a little, I decided to provide it here as a blogpost. I think it’s a decent example of how one can not only build useful abstractions for pure problems in Haskell, but also for very imperative-like and impure code. Additonally, I vaguely recall people on #haskell saying we need more tutorials on real-world stuff.

The problem is quite simple: a component generates a number of requests to a REST API running on another machine. We need to implement this in a fail-safe way: basically, the other machine or the network can go down for a while. In this case we should cache all requests and try them again later (but in the same order).

Since this is written in Literate Haskell, we first have a few imports you can safely skip over.

Let’s step away from the problem of making REST API calls and come up with a more general, informal description: we have some sort of “action” which runs or fails. If the action fails, we need it try it again later.

We want our queue to be thread-safe. For this purpose, a simple MVar will do. Our queue will be represented by a Sequence. We could also use a simpler queue data structure as given by Okasaki, but we’ll stick with Sequence since it’s in the commonly used containers package.

And so we define our Queue: an MVar for thread-safety around our actual queue.

People unfamiliar with the Haskell language might be confused at this point: we used data, type, and newtype – three different ways to create a type! Let’s elaborate on this a little:

data creates a full-blown data type. Using data, you can create constructors, records, all of it;

type simply creates a type alias, it does not actually create a new type. Hence, type is mostly used for readability reasons: we can now write Retryable instead of IO Retry (while they still mean the same thing to the compiler!);

newtype creates a new type, with the restriction that it is a wrapper around another type. This holds advantages from a type-safety point of view: we cannot accidentally mix up types, and the implementation of the Queue can be hidden from the user. newtype wrappers are optimized away at compile time, so introducing extra type-safety has no performance penalty!

We modify our thread-safe variable by appending (|> is appending to the right side of a Sequence) our new Retryable. Then we immediately try to pop from the queue. This final pop gives us the nice property that all actions are performed immediately after being pushed in an everything-works-fine-scenario.

The definition of pop holds the main logic for our queue-based system, and it’s defition looks more complicated, mostly because we need to handle all different cases: