2013-05-20: Efficient Forth in 85 Lines of Racket

Forth
is an interesting programming language in the concatenative
family. It relies crucially on a data stack, rather than functions
with their arguments and return values, as a means of communication.

Last week I wondered, "Could I implement a Forth-like system in
Racket, with interoperablitly between the two, with macros?" This post
presents
the
system.

-

1Basics

In Racket, when you want to add two numbers, you pick the numbers and
deliver them to a function that does the addition and returns the
result:

In most languages, functions are restricted to returning a single
value, like 3 in this example. However, in Racket we do not
restrict the arity of continuations and allow you to return
many values to the current continuation and create a context that
expects many results:

In Forth, rather than than picking the numbers and delivering
them to the function, you put the numbers in a place the function can
get to them. Similarly, rather than the function returning the answer
to you, it places the answer in same shared location where the
rest of your program can do something with them.

This shared location is the data stack and is the fundamental
thing that Forth programs revolve around. A test case for a Forth
program is a triple of (1) an input stack, (2) a program, and (3) a
result stack.

In this example, simply by writing a number, like 2, we are
placing it on the stack, and by simply writing a function like
:+, we are jumping to it where it can do its work.

In some sense it more natural to write the second example, because we
can more directly place multiple "answers" on the stack. On the other
hand, it is more complicated because we have to explicitly copy
information that we don’t want to destroy. In Forth, you can do this
with functions like :dup that duplicate the thing on
top of the stack and :over that duplicate the thing just
underneath that.

2Specification

We would like to enable this sort of programming in Racket, but with
the additional feature that Racket code should be able to seamlessly
call Forth code, provided we specify how many stack locations
the Forth code touches. For example, we wouldn’t be able to call
:f, because it is not obvious that it expects two things on
the stack and adds one. But, we could specify that and then call it:

We must be able to define functions in Forth that are callable
from Forth, always.

We must be able to give functions stack effect annotations to
enable them to be called from Racket.

We must be able to lift Racket functions to Forth so they
are oblivious to the stack, like turning + into :+.

We must be able to lower Racket functions to Forth so
they can directly affect the stack, like writing :over.

We must be able to enter Forth from Racket arbitrarily, such as
to write testing forms like check-forth.

3Implementation: Data and Tests

The first thing we’ll do is decide to represent the stack as a normal
Racket list. Next, we’ll represent Forth functions as special data
structure that holds a function that accepts a stack and returns a
stack:

5Implementation: Definition Macro

The forthda macro uses some advanced macrology and is the
crown jewel of this implementation. Its basic structure is to expand
each forthda into a stack transformer function, a new
sub-struct of forth-word, and an instance of that sub-struct
with the given stack transformer:

Even at this level of abstraction, there’s a few tricks. First, we use
~or to combine the three different kinds of
definitions. Second, since the identifier used for the stack is only
required in the #:lower case, we give it the default
#'stack and use a different name for the combined
case. Third, notice that the stack effect specification is sometimes
required and sometimes optional. Finally, we rely on a definition of
the stack-spec syntax class:

If we are lowering, then we just inject the body, otherwise we use the
given stack argument and hand everything over to the forth
macro to do the association and evaluation.

The body of a lifted function is more complicated, because we need to
read a fixed number stack positions, call the Racket function, get
back a fixed number of results, then put them back on the stack. We
could use something like take or split-at for that,
but it would be expensive, since we know exactly how many positions
are needed at compile-time. In addition, we’d have to use
apply and call-with-values to invoke the Racket
function, which is also expensive.

Given that we know those two numbers, we can generate the appropriate
number of temporary identifiers and then put them into two lists: one
going forward and one going backward. This enables the following
lifted body definition:

We read the input of the stack backwards, then call the function with
the input forwards, then read the results forwards, and put them on
the stack backwards. For the lifted definition of :+, this
looks like:

Finally, we have to define the struct for this forthda. The
key here is that when the function has a stack specification, then
give the new struct the prop:procedure structure property, so
Racket code will treat it as a function.

However, we have a dual problem to lifting: Racket will give us a
finite number of arguments, which we need to put into a stack and then
call the stack transformer, and read off a fixed number of answers.

And, of course, if there is no stack specification, we don’t give it
the property, so Racket code can’t call it: