Tail Call Optimisation in Common Lisp Implementations

While the Common Lisp standard does not require implementations to
eliminate tail calls, some do—thereby allowing a more functional
programming style. Especially in the prototyping phase, it can be
useful to know where this functional style can be employed and where
it's likely to cause stack overflow. Since I couldn't find a single
document outlining support for tail call optimisation, I've written
it.

In what follows, I distinguish between Tail Call Optimisation
(TCO)—where all calls in tail position are optimised into
jumps—and Tail Self-Call Optimisation, a special case of TCO where
only self-calls are optimised (this is usually what people want when
they refer to TCO as it allows for the recursive style). TCO is
sometimes referred to as proper tail recursion.

This document is still incomplete; feedback would be appreciated
(especially where claims are just outright wrong!).

Nice to haves: [0/2]

[ ] Demonstration of trampoline code;

[ ] Discussion of optimisation declarations and their effects on
each implementation.

2 Implementation Details

2.1 DONE Cmucl

2.1.1 Documentation

Both the compiler and the interpreter are ``properly tail
recursive.'' If a function call is in a tail-recursive position,
the stack frame will be deallocated at the time of the call, rather
than after the call returns.

"Properly tail recursive" refers to all calls in tail position
as is made clear below.

2.1.2 DONE Demonstration

Using: CMU Common Lisp 20b (20B Unicode)

Let's take a look at some disassembled functions. First up, tail
recursion:

Indeed, CLISP implements the tail call optimization. To see one
difference between them, we can run the two factorial functions on
some large input (say 4000). While factorial-rec will fail with a
stack overflow, factorial-iter won’t – this is because the stack
does not grow.

Unless directed otherwise, the Compiler optimizes self-recursive
function calls, tail calls, and self-tail calls. In particular,
self-tail calls are automatically compiled as loops. While these
function calls are efficient, they can be difficult to trace because
they do not appear on the stack.

and

When the Compiler compiles either a tail call or a self-tail call,
it reuses the calling function's stack frame rather than creating a
new stack frame. This optimization is called tail merging.

If true, compiler will perform self-tail-merging (for functions
which call themselves). This switch will affect code only when true
at the beginning of the function being self-called (and no other
time)….

If true, compiler will perform non-self-tail-merging (for functions
in the tail position different than the function being
executed). This switch will affect code only when true at the point
of the call.

Initially true if speed is greater than 1 and debug less than 2.

2.6.2 Demonstration

Allegro 8.2 64-bit, OSX.

By default alisp does not produce the fastest possible code in the
interpreter,

2.9 DONE Armed Bear Common Lisp

2.9.1 Discussion

This is a well known JVM limitation (you have to resort to loops and
trampolines). The Common Lisp standard does not require TCO, but
every other Lisp seems to do it, and I, coming from ML and Erlang,
am used to programming in a very recursive style. But this is just
me, orthodox lispers will tell you that you have to use loops.

2.9.2 Documentation

As described elsewhere, the LispObject is the base component of
which the Lisp world in ABCL has been built. Everything is a
LispObject. LispObject has 10 methods which might get called when
evaluating that "Lisp object": they're all called execute and
implement the zero - nine argument forms and the array argument
form (which allows for non-predetermined numbers of arguments).

When an object is evaluated one of the appropriate forms of the
execute method is called. Code deeply nests calls to execute()
methods, because the evaluation of a function inherently calls
execute() methods to call other functions.

This system leads to large stack sizes: Java doesn't allow tail call
elimination.