Archive for August, 2012

Christopher Vollick and I were recently discussing ways to get more compile-time guarentees out of our web templates. We both love Mustache for its simplicity and its thin-view philosophy, so that was a natural place to start.

I decided to try writing a tool to convert Mustache templates to Haskell code, which I have done.

This allows the compiler to check a number of things for us including: ensuring we do not use any keys in the templates that are not defined in the model, ensuring that we do not use keys in the template in ways that are inconsistent with the datatype in the model, and ensuring that our partials and sections expect the context they will end up getting passed due to the structure of the templates.

It also gives us a speed boost. Comparing to the other popular Haskell Mustache implementation my current benchmarks show a significant speedup. This is to be expected, since GHC can perform optimisations that a runtime interpolator just can’t. Also, in order to be able to use arbitrary Haskell Records as context, runtime implementations have to perform runtime checks using a union type or the Typeable class. My implementation can do all these checks at compile time.

People familiar with Mustache may be wondering how I handle recursive partials, since this is a feature Mustache touts as being possible because the templates are not compiled. I make a compromise in my code: partials are rendered out as toplevel functions, just like any other file, and as such they only inherit the context directly above them, instead of inheriting the enitre scope all the way to the top. I rarely use values from higher scopes in my partials anyway, so I don’t think this is a very severe limitation.

Values that are displayed in the template must have an instance of Pretty from wl-pprint and values that are used in sections or inverse sections must either be lists of records (which get used as the context for the section body) or any Monoid instance. If the value (==mempty), it is considered falsey. Note that if you need values to be able to be optional, but want the natural mempty of your datatype to be truthy, you can acheive this by wrapping in a Maybe.

The requirement to be a Monoid is relaxed for Bool and a large list of numeric types, which get wrapped in the Any and Sum newtype wrappers by the code generator when they are detected in a record. This allows False and 0 to be treated as falsy for the majority of types where this would be interesting without needing to define orphan Monoid instances or wrap everything up in newtypes yourself.

Last time we got hardware interrupts working, and used them to implement preemption for our multitasking. This time we’re going to get our user space processes talking to one another.

getpid

We’re going to need some way for replies to our messages to get sent back to a process. We can’t use whatever other mechanisms we build for registering a new communications channel for this, because without it we cannot receive communications, even about a new communications channel. Luckily we have a piece of information that comes with every process: its ID. In our case, the ID of a task is going to just be its index in the array of stacks in our main.

Putting processes to sleep

Currently our scheduler just round-robins through all tasks, but if we want a task to be able to wait for a message, we need some way for it to go to sleep while it waits. We will do this by having a new piece of metadata in our process space for “status”:

Ringbuffers

We’re going to do our communication between processes using a rudimentary implementation of UNIX-style pipes. The pipes will be implemented using ringbuffers. Why? Because ringbuffers are easy to use for FIFO-style communication without needing any dynamic memory allocation. Pipes are allowed to get “full” and luckily we can tell when a ringbuffer is full, so this fits. We’re going to want some generic ringbuffer utilities, and it won’t hurt to have versions specialised to the pipe buffers:

You’ll note I’ve increased the stack size. We’re going to be handling more data, and with our current setup if you need more space than the stack size it’s just going to cause very strange and hard-to-find bugs. The pipe limit is expressed in terms of the number of tasks. This makes sense since as the number of tasks increases, it is likely that the number of channels we’ll need increases, especially since we’re going to reserve one per task to begin with.

You’ll note that I’ve placed the bodies of these syscalls off in their own functions. That’s partly because they’re going to be much bigger than what we’ve seen before, but also because they actually need to be able to call each other.

Loop over all the tasks, and any that were waiting to write, unblock them and try the write again. This is somewhat inefficient, since only tasks that were waiting to write to the pipe we just read from have any chance at all of being able to succeed (all the other pipes are just as full as they used to be), but it works.

Again, an out-of-range pipe index is obviously an error. And because of the ringbuffers, you’ll never be able to write more than PIPE_BUF bytes into a pipe all at once. We want our writes to be atomic (because we have preemptive multitasking going on), so trying to write more than PIPE_BUF bytes needs to be an error as well.

Pathserver

So, we have working read and write calls now, but in order for processes to communicate they’re going to need to agree on what pipe indices to use. A parent process can get its child’s PID (from fork) and write to it that way, maybe, but that’s not a generic solution.

We’re going to solve this problem by building a nameserver. The names in question are going to be pseudo-filesystem “paths”, so let’s call it a pathserver. We’ll need some string functions:

We said we’re reserving one pipe for each task for replies, and let’s also reserve 0-2, since those file descriptor numbers are usually “magic” and we don’t want to have any mix-ups until we get real FD tables in place later. The pathserver will manage all other pipes, so it can assign itself the first one. Now we have a constant that tells us the pipe index of the pathserver.

We can use this information to define a protocol and pseudo-syscall for registering a new path:

Notice I said pseudo-syscall. It may act sort of like a syscall, but in fact it is built entirely based on primitives we already have! This is one of the huge benefits of a microkernel: once you get your basic functionality in place, most things can be built on top of what you already have, as wrappers.

The protocol here is a zero, followed by the length of the path we’re registering, followed by the characters of that path. Pretty simple. There’s no way for the pathserver to signal an error to us, because we don’t wait for a reply, but unless there are no more pipes this can’t fail, so we’ll just leave it for now.

The protocol here looks the same as before, only instead of 0 as the first word, we send our reserved pipe. That’s our PID, moved ahead by the three we’re reserving. Conveniently, this means no valid replyfd will ever be 0, so we can differentiate between mkfifo and open.

Loop forever, reading the three pieces of data out of our pipe. Recall that we wrote all of this data with a single write call. That’s important because we need all these bytes together, so the writes must be atomic. Once we’re reading, however, the bytes are already in a deterministic order, so we can read them out in pieces. This is especially good, because we wrote the length of the message as part of the message! So we certainly need to be able to read one piece at a time.

if(!replyfd) { /* mkfifo */
memcpy(paths[npaths++], path, plen);

Registering a new path is super easy. Should probably check for out-of-bounds here, but this works for now.

Linear search for the path. If we find it, adjust the index by the reserved indices and write the reply. We ignore empty paths, to allow us the possibility of pipes without names that the pathserver knows about, but you can’t usefully search for. Notice also that we set i back to zero after writing a reply, because:

We spin up the pathserver and the otherguy. The otherguy just waits for incoming strings (19 characters max, but it’s just a demo) and prints any that come in. Then, our first task run in a loop forever sending Ping to the otherguy, and they get printed.

We just built a rudimentary serial port driver! It’s not using interrupts properly yet, but at least strings send to the otherguy are printed atomically, because the otherguy “owns” the serial port and we don’t have all the tasks preempting each other in the middle of a write.

I think Ubuntu inherits the things that need to be configured for this to work directly from Debian; if that’s correct this information will be useful without modification for Debian systems. It may also be possible to configure dhclient on other Linuxes (and other *nixes) according to these notes; I don’t know whether the guts of dhclient-script come from Debian or from dhclient. In any case, looking at what dhclient-script does will be a good start.

So, I have a machine that’s attached to two networks.

Both networks have DHCP servers. Both DHCP servers give out router configuration. But one of those default routes is useless, so I want to force the system to always use the other network segment’s router as its default route.

(My use case is a virtual machine; one VM network is NATted to the outside world, and one is isolated on the host machine. This VM gets run on different host machines, which assign different network addresses to their virtual segments, but which interface is on which type of segment is fixed by the VM configuration. So here, unlike many multi-interface machines, it’s necessary to tie the default route to a particular *interface* and not a router address.)

It turns out that this isn’t hard to do, but it took a lot of googling (and several rounds of “Find something interesting, search on new keywords”) to find a solution that actually did what I needed; all of the obvious searches turn up information about how to set the default route to a fixed router IP, which only works when you’re always on the same network.

Before I get into the theory of operation, here’s the solution, for
impatient people:

So, here’s how this works, since that might be useful for other complicated DHCP configuration tricks you want to do.

When dhclient needs to make system configuration changes, it does it by invoking /sbin/dhclient-script. The first nontrivial thing dhclient-script does is run the entry hooks in /etc/dhcp/dhclient-enter-hooks.d to allow local configuration rules to adjust things as necessary.

These hooks are shell script fragments sourced by the main script, so you can use them to edit the variables that make dhclient-script do things if you like. This is precisely what we want to do. $new_routers is the default gateway the DHCP server gave us; we only want to actually use it for the DHCP server eth0 is talking to, so if we’re on any other interface we unset it, to prevent dhclient-script from taking any action based on it.

You can use the same trick to ignore DNS settings (new_domain_name and new_domain_name_servers) if you want to ignore one network’s DNS servers; I didn’t need that in my configuration because the two subnets’ nameservers behave identically. (But note that if you need something more complicated than simply ignoring one of them, you’ll probably need more than just an entry hook.)

One of the things statically typed languages like Haskell and C often don’t have great support for, is collections that contain elements that are not all of the same type. In a dynamically typed language one can store any element of any type in a collection, because the type is finally determined later when the value is used at runtime. With a statically typed language, the compiler wants to determine all the types at compile time, and so it often wants to force every element to be of the same time (homogeneous collections).

Normally when this problem arises, I find that I can rethink the problem in a way such that the solution does not require heterogeneous collections, but recently I’ve been working with one that, while I can solve it another way, the most natural solution seems to call for this.

The Haskell Wiki has a couple of suggestions for doing this. The easiest one, is just to use Data.Dynamic to do dynamic typing from inside Haskell. The problem with this is that it is both slower (the types of all values in the collection have to be checked at runtime) and less safe (if a bad value gets put in the data structure, it can cause the application to crash).

Another solution suggested there is exitstentially qualified datatypes. These solve the problem in a way that is safe (as long as you only need to perform operations from some typeclass on the data) and efficient, and is analogous to an upcast in a statically-typed Object-Oriented language. However, it can be a bit complex, and is a language extension.

A solution that will work in Haskell98 and give most of the benefits of exitstentially qualified datatypes (with less power) is to package up operations to be performed with the datatype, instead of the data itself. Any functions that you will want to call (such as those which are members of the typeclass you’re interested in) can be partially applied to the data, resulting in a function that is the exact same type as an equivalent function partially applied to some other instance of the same typeclass. You can thus put records full of these functions into a list or other collection, and it will work because they are all the same datatype. Then instead of calling (func x otherparams) you can just call (func otherparams) and get exactly the same result.

Why prefer code generation over template Haskell? Isn’t them essentially the same thing, and template haskell is performed automatically.

Also, from Reddit (nicolast):

I never understood why someone would want to avoid using language extensions which have been in GHC for at least some time. The only reason I can think of is: compatibility with other compilers. But is anyone ever going to compile/run a yesod-routes based application using something other than GHC?!

First off, yes, Template Haskell is very similar to code generation. There are a few reasons I would like to avoid it.

It’s a language extension. I try to avoid those in general, in every stanardized language I code in (C89, R5RS, Haskell98) for several reasons.

As nicolast said, compatibility with other compilers is a big one. When I get a piece of code from someone who assumed that “what MSVC does” or “what Racket does” is the same as “anyone can run this”, it makes it quite difficult to use my favourite implementations of those languages. I don’t want to make assumptions about other people’s environments, or what will be useful in the future. Maybe someone writes a Haskell interpreter that makes use in some context I haven’t even imagined much nicer. Who knows.

Additionally, any other static analysis/code processing tools (like, say, hlint) *also* needs to support whatever syntax extensions you’re using (semantics extensions may or may not apply here, depending on the nature of the tool). Requiring that every tool author support all my favourite extensions limits my tool options, and makes life harder for tool authors (since they cannot just look in one place for the spec and write to that anymore if they need to look up compilers’ extensions as well).

Will anyone ever compile/run/analyze a yesod-routes based application using something other than GHC/hlint? (Actually, does hlint support TH? It might.) What specifically about yesod-routes makes this less likely? What drew me to Yesod.Routes.Dispatch was its relative purity in terms of extensions/dependencies, etc.

Additionally, I find Template Haskell specifically (and some other language extensions, like Overlapping Instances) can make code harder to read (for me) and possibly harder to reason about. A code generator makes a file that I can read for comprehension, edit if I want to, etc.

Ok, that’s a bit of a long answer to a short question, but it sort of sums up my motivation vis extensions in general and TH in particular.