Links

Multipass Rendering With Mustache

I love Mustache for rendering. Its simplicity
pretty much forces you to do The Right Thing™ in code that uses it, and
it's got an implementation in just about any language you might find yourself
using. That being said, I did find myself wishing for one feature:
multipass rendering. So, loving both Mustache and Ruby, I decided to add support
for multipass rendering to the Ruby implementation of Mustache.

What's multipass rendering?

Multipass rendering is a term often used in relation to 3D rendering. It refers
to splitting the rendering work into multiple parts, or "passes", for any of a
number of reasons. One of these reasons, and the one which is most likely to
apply in the area of template rendering, is the case when some rendering passes
change while others stay the same.

For instance, consider the following template:

Hello, {{ name }}!
Multipass rendering is {{ adjective }}!

Now, if you want to render this template once, it's easy enough:

Mustache.render(template,:name=>'Ernie',:adjective=>'awesome')

Hello, Ernie!
Multipass rendering is awesome!

But what if you want to render the same message hundreds or thousands of times,
while only changing the name? Sure, you could render the entire message over and
over again, but that gets increasingly expensive as the template grows. Besides,
what if you wish to split the rendering up across different apps or machines,
not all of which share access to the same contextual data?

Hello, {{first_name}} {{last_name}}!
Multipass rendering is awesome, and you are {{adjective}}!

So, this isn't what you want. You've lost the tags that "step in" to person,
and you'll need to make the values include a person prefix as a result. And
then you find you want to allow triple-stache (unescaped) values for some
attributes, and the whole thing falls apart again.

This isn't working.

It's just Ruby (in Ruby)!

It's time to look at the Mustache source. Digging around, we find
Mustache::Context#find, which looks like this:

Sweet. So it turns out that a Mustache context figures out whether an object in
its stack has a value by seeing if it looks "hashy", and if so, treating it like
a hash. Otherwise, it falls back to sending messages to Ruby objects. We can use
this knowledge for good.

What if we had an object that had any key we could ask for, and returned the
"right" value for that key, according to how it was being used in the Mustache
template? Well, it wouldn't be that easy, since the object doesn't really know
how it is being used. When you call Mustache#render, Mustache converts the
object passed as the first parameter into a Mustache::Template, which is what
actually does the rendering. If we take a look at Mustache::Template#render,
we can see how this happens:

defrender(context)# Compile our Mustache template into a Ruby stringcompiled="def render(ctx) #{compile} end"# Here we rewrite ourself with the interpolated Ruby version of# our Mustache template so subsequent calls are very fast and# can skip the compilation stage.instance_eval(compiled,__FILE__,__LINE__-1)# Call the newly rewritten version of #renderrender(context)end

Mustache "compiles" the template into a string consisting of Ruby code which
accepts a context and returns the actual output. It then defines a singleton
method named render on itself, which is the result of this compilation step,
so the compilation step doesn't need to be repeated for this template, and calls
this newly-defined method.

Mustache::Generator creates the aforementioned string of Ruby source from a
tokenized version of your template, stepping through the tokens and firing
events as they are encountered, SAX-style. This happens in
Mustache::Generator#compile! (the root expression is a :multi containing an
array of other expressions):

What's it do? Well, it expects an initial "prefix", which serves as its string
representation, and it returns a new LaterContext for any key, appending the
key's name to its existing prefix using Mustache's dot separator.

You may also be familiar with Mustache syntax like:

<ul>
{{#people}}
<li>{{.}}</li>
{{/people}}
</ul>

In Mustache templates, a bare dot like this refers to the object that's been
"stepped into" via a section. So given the following context:

{:people=>['Bob','Jim','Tom']}

The rendered template would be:

<ul><li>Bob</li><li>Jim</li><li>Tom</li></ul>

In Ruby, the dot is essentially an alias to the object's to_s method (or key),
because the Mustache parser creates "fetch" expressions that split the value
inside the key on the "." character. When the generator encounters an empty
array to fetch, it returns the Ruby code: ctx[:to_s] (where ctx is the current
context in the stack).

So, we would really like to handle that case as well. We need an object that is
a string (so it can return itself on to_s calls or keys) but can be
differentiated from any other "." by the generator. We'll add it to the stack
any time we step into a LaterContext via a section. Let's call it
DotContext.

By returning a new LaterContext when keys are retrieved, this should allow us
to generate non-prefixed tags inside sections.

With those classes out of the way, it's time to do some Mustache subclassing!

M-M-M-MULTISTACHE!

Since we're subclassing Mustache to do multipass rendering, what better name
than Multistache? I'll save you the time: there is no better name.

We'll set up the class method that "templateifies" objects to do so using a
to-be-created subclass of the normal Mustache::Template class, and we'll add a
factory method for creating a LaterContext while we're at it.

All of those pieces don't mean anything if we don't have a generator that knows
what to do with our new objects. This one is going to look a lot hairier than it
really is. Let me explain what we need to do here.

So, here's Mustache::Generator#on_etag, which handles generating code that
returns an HTML-escaped string, such as {{name}}:

It's pretty straightforward, though the parameter name is a little misleading.
Here, name isn't actually a name, but a parsed expression like:

[:mustache,:fetch,["name"]]

Earlier, we looked at the compile! method. I didn't mention the :fetch type
of Mustache tag then, but there it is. So, as you might imagine, there is also
an on_fetch method which handles this case, and takes care of returning code
that will fetch the attribute from the context. In this case, it will
generate...

ctx[:name]

...which does about what you'd expect, returning the result of reading that key
from the Mustache context using a hash-like syntax. The context takes care of
searching through its stack to return the proper value.

So, then, the generated code from the default on_etag method would look like
this, in the case of {{name}}:

In other words, if the value returned from the context is a Proc, we want its
return value to be treated as our special kind of template, and not the default
Mustache one. If the value ends up as a LaterContext or DotContext, we want
to render a string that corresponds to the Mustache syntax for an "etag", to be
rendered in a different pass.

The other 3 tag types are handled similarly, with the interesting part being
that, as mentioned, when we step into a section, we push a DotContext onto the
stack, so that we don't actually render any attributes inside the section. This
is because we don't have the necessary information about the object's keys in
order to know whether or not any given attribute would be fetchable from the
deferred object or would fall further down the stack. An inverted section has
no such requirement, since it doesn't step into the object in question.

It's unfortunate that in order to get this behavior, we have to duplicate so
much of the existing Mustache code, but because these methods are generating
code, and we need to modify behavior inside the middle of that generate code,
it's not just a matter of calling super. Bear in mind that while this code
listing is kind of long, the only changes from stock Mustache are the kinds I
outlined above.

Hello, Ernie Miller!
Multipass rendering is awesome, and you are nifty!
Hello, Joe Blow!
Multipass rendering is awesome, and you are lame!

And there you have it. But this post wouldn't be complete without a few...

Caveats

This all works really great, but keep a few things in mind:

If you're deferring rendering of sections, then any contextual references
contained inside the section will be deferred until the second pass, for
reasons explained earlier. This means if you refer to contextual information
that won't be available in the second pass, you're gonna have a bad time. In
practice, this isn't generally an issue, but it's important to keep in mind.
If your second rendering pass has a context that is always a superset of the
first pass, you can disregard this warning altogether.

Remember that an inverted section will have its content rendered on the first
pass, and only the {{^key}} {{/key}} tags will be left unrendered. If you
are referencing contextual data that isn't available until the second pass,
you'll want to supply LaterContext values for them, to defer rendering.

You'll note that the name of the context key is specified as the parameter to
Multistache.later. Yes, this means you can actually rename a contextual
reference on the first pass for use in the second. I have no idea why you'd
want to, but it's kind of cool.

Wrapping up

I hope you found this journey through the innards of Mustache interesting, and
the code examples useful. By the way, all of the code is bundled up for you in
this gist.