Three versions of the routine implementing call evaluation for user defined functions ("expr" in lisp jargon)
are detailed here. Starting with no optimization and continuing with the tail recursion optimized version
helps to explain the full version implementing mutual recursion optimization.

No recursion optimization

This is the basis. The call to bind evaluates arguments, stores current values and updates the environment with new values.
The body of the function is evaluated with the eprogn in the correct environment.
The try .. finally ensures that the environment will be restored whatever happens.

Note that the try ... finally construct is equivalent to the
following try ... except ... else. This enables to have a special
process if required for some exception. This is put at work in mutual recursion optimization.

The eprogn is embedded in a while loop and
a try ... except. If no recursion is detected, the break
stops the loop and the execution
flow continues as usual. However, if a recursion is detected, the TailRecException is triggered
and caught in the except clause of the previous call. This by-passes the break and makes the loop to continue,
explicitely tranforming recursion into iteration.

Suppose we are evaluating a tail recursion call in 2.01. This call must be the last expression of the eprogn at 1.11. At this point, the bind record is on the stack and variables have they new value (2.03).
When the tail recursion is detected (2.04), the binding record is removed (2.05) and the tail recursion exception is raised (2.06).
The exception is caught at 1.13. Being in the loop (1.09), execution returns to the eprogn (1.11).

Mutual recursion optimization

Mutual recursion optimization requires a little more mechanics to be handled:

The identity of the function must be tested to apply optimization at the right place.

The environment of the intermediate calls must be preserved. This is done by merging the
binding record with the one of the previous call.

Suppose we are evaluating a mutual recursion call in 3.01, i.e.
f12 in (f11 ...) --> (f2 ...) --> (f12 ...).
This call must be the last expression of the eprogn at 2.11.

When the mutual recursion is detected (3.04), the binding record is removed (3.05) and the tail recursion exception is raised (3.06).
The exception is caught at 2.13. Being an intermediate call (f2), the bindings are merged at 2.17 and the current
binding record is popped. The exception is reraised at 2.19 and finally caught at 1.13.
This time, the test at 1.14 is true and this is handled at this point
as tail recursion.

Note

The current implementation adds an argument to evalexpr and
bind
routines to specify whether function arguments are evaluated or not. This enables to use the same routine for
eval and apply.

Tail recursion detection

The tail recursion calls are detected in the history of calls. This history is a stack and it is handled explicitly in sapid implementation. This stack contains various types of records. The ones uses to store arguments are tagged with the keyword lambda. These records contain the definition of called functions and this enables to compare the function being evaluated and the calling ones.

The tail recursion is detected by comparing the top function value to the previous one. The exception mechanism will clean the host stack and the top record is removed from the evaluation stack. It is clear that there is no need to restore the value of variables x and y to 4 and 8 if they are immediately replaced with values 5 and 7.

The same mechanism is used for mutual recursion with the precaution that intermedate records must be merged with the first record of the recursive function.

When the recursion is not terminal, something must be in added in the stack to avoid detecting a tail recursion. This is the role of stack records of type eval.

It is clear that saving a value (4 for instance) is not necessary if the previous value (5 in that case) will override it immediately. This optimization is not available in current implementation. It could be extended to mutual recursion and to any number of subr calls between calls of the recursive function.

Timing

Running the test suites defined in tests.l enables to estimate the computation overhead needed by the optimizations presented here. Note that the interpretor can be configured to handle no recursion optimization, tail recursion optimization or mutual recursion optimization.The next table gives the figures obtained with the following conditions:

Pentium 4, 3.2 GHz, 2 GB RAM

each suite repeated 10 times

svn revision 80

Timings are given in seconds.

Configuration \ suite

no optimization suite

tail optimization suite

mutual optimization suite

no recursion optimization

160

cannot be processed

cannot be processed

tail recursion optimization

180

89

cannot be processed

mutual recursion optimization

181

91

64

With the usual precautions, the conclusion is that the technique presented here have a cost of about 12% compared with running without optimization. This is quite acceptable given the benefits of being able to process any size of data when using tail and mutual recursion.

References

Recursion optimization is usually done by simulating in some way a stack machine. This is of course powerful but leads to code far from the natural non recursive writing. Playing with exceptions and after some experimentation, I got the tail recursion version and managed to extend it to mutual recursion optimization.

The technique seeming unusual, I have spent some time googling for a precedent. Actually, a similar technique is used to implement python decorators transforming tail recursion into explicit loop. For instance, this decorator has a structure very similar to the one used in sapid lisp.