Compiling queries without EVAL or COMPILE

What he should have done is written a compiler. Luckily Common Lisp
comes with a Lisp compiler built in so all we have to do to write our
own compiler is translate our source language into Lisp.

I agree that Gary should have used a compiler, but there's a
problem with Peter's alternative: the Common Lisp compiler is
usually quite a heavy thing to invoke.

Gary's example problem can be solved without using EVAL or
COMPILE. The trick is to produce a tree of closures that implement
the semantics of the
expression. cl-ppcre, for
example, uses this strategy to great effect, and SICP also
covers it in detail.

What does it mean to compile to a tree of closures? It means that
for a given form, a closure is produced that calls a particular
subsequent function if the form is successful (whatever that
means) and another subsequent function if the form fails. The
user is given the root of the tree, and invoking it with a target
string will work through the functions in the tree until a success
or failure function returns.

This can minimize the work by using the short-circuiting semantics
of operators like AND and OR, without using AND and OR directly.

Here's Gary's example expression:

'(or (and "space" "mission")
(and (or "moon" "lunar") "landing")))

Here's a dispatching function that looks at a form and calls a
specific closure creator depending on the kind of the form:

When these functions are all compiled, they each produce compiled
closures. In contrast to Peter's solution, COMPILE is used only
once, possibly in the distant past, to produce the closure-creator
functions.

It's easy to add more operators. For example, here's the closure
creator for a NOT operator:

This result is partly because SBCL has a pretty slow
compiler. Other compilers vary, but they're all between 100 and 1000
times slower at COMPILE-EXPRESSSION.

MAKE-MATCHER is pretty simple. In a more flexible system, you could
move work out to generic functions that specialize on the form type
and the operator of the form. Chris Riesbeck talks about this
in Graham
Crackers, using Graham's raytracer as an example.

As Peter mentioned, it would also be helpful to keep track of which
strings have been seen, and their presence or absence in the target,
to avoid making multiple tests.

The main point, though, is that avoiding EVAL by using COMPILE is
partly an exercise in shuffling a big cost from runtime to a smaller
cost at compile-time. The closure creation scheme not only matches
the run-time performance of COMPILE, it reduces the compile-time
cost as well, sometimes dramatically. When compile efficiency is
important, it's a handy tool to have in the toolbox.

If you want to experiment with the different
approaches, gwking.lisp
includes the code from Gary, Peter, me, and adeht, along with a
little bit of benchmark code.