Staapl

1Introduction

Staapl is a Scheme to Forth metaprogramming system. To illustrate the
general idea we’re going to use a concrete application: the Forth
compiler for the 8-bit Microchip PIC18 microcontroller architecture.
This section uses REPL interaction with some example code written on
top of the compiler to demonstrate the code generation process.

1.1Forth

The code> form provided by the demonstration module
interprets the forms in its body as PIC18 Forth code, compiles them,
and prints out the resulting intermediate code with one instruction
per line.

> (code>123)

[qw 123]

The instruction qw tells the target machine to load the
number 123 on the run-time parameter stack. It is short for
“quote word.” Providing a sequence of numbers in the code>
body will generate concatenated machine code, which is executed by the
target machine from top to bottom.

> (code>123)

[qw 1]

[qw 2]

[qw 3]

Target code is represented in this intermediate form during the first
code generation pass to facilitate code transformations. It consists
of a mix of pseudo instructions and real PIC18 instructions. The code
generator will eventually clean up all occurances of qw
before attempting translation to binary machine code. The
pic18> form performs this extra step and shows real machine code
output.

> (pic18>123)

[movwf PREINC0 0]

[movlw 123]

The first instruction stores the contents of the working register in
the 2nd position on the parameter stack, and the second instruction
replaces the contents of the working register with 123.
Again, concatenating compiler input produces concatenated output:

> (pic18>123)

[movwf PREINC0 0]

[movlw 1]

[movwf PREINC0 0]

[movlw 2]

[movwf PREINC0 0]

[movlw 3]

The intermediate instruction set which contains the qw
instruction is useful for implementing partial evaluation
rules. When compiling a particular Forth word, the compiler can
inspect the code already compiled to determine if it can combine its
effect with the effect to be compiled.

This illustrates 3 different modes of computation. The first program
computes the addition at run-time, taking both input values from the
runtime stack and putting back the result. The second program adds
the literal value 1 to the top of the stack using a different
machine instruction. The third program doesn’t perform any run-time
computation at all and simply loads the result of the addition that
was computed at comple-time because both inputs to the addition where
available.

Note that in this last program the result of the compile-time
computation is not shown. Instead it shows a program (12+)
that gives the result upon evaluation. The compiler doesn’t need to
know the exact value at this point. It only needs to know that the
value can be determined later when necessary. This is essential for
integration with the assembler, since these expressions might contain
symbolic representations of code addresses that only the assembler
knows.

Using the intermediate form with the qw pseudo-instructions
to compile the Forth program 12+ shows the key idea: the
target code list can be interpreted as a parameter stack, with
the top of the stack at the bottom of the code list.

The code stack can be used as the argument passing mechanism
for a language of macros that is active at compile time.
Machine instructions then become datatypes of this language. The word
+ names a function that operates at compile time.
It inspects the code stack and if it finds one or two qw objects it
can use them as input to the addition operation and compile a simpler
run-time instruction.

1.2Scheme

While traditional Forth has its own metaprogramming facilities,
combining it with a Scheme-based meta system gives the added advantage
of tying into a powerful lexical scoping mechanism. The
ability to assign local names to objects comes in handy when dealing
with complex data structures, which is sometimes difficult to do in a
combinator language like Forth or Joy. At the same time, the absence
of lexical binding forms in Forth code make it very suitable to be
handled as data, avoiding complications associated to name
capture.

This sections introduces the forms macro:,
compositions, patterns and tv: which
comprise the metaprogramming interface between the Forth and Scheme
languages in Staapl.

1.2.1Code and Composition

A representation of a Forth program can be composed using the
macro: form.

The objects created by macro: are called concatenative
macros. These are functions that operate on compilation state
objects.

> code1

#state->state

The target code associated to the objects code1 and
code2 can be printed using the function
state-print-code which extracts a code stack from a state object
and prints it, and the function state:stack which produces a
state object with an empty code stack to serve as an initial state.

It is seldom necessary to apply concatenative macros to state objects
manually since composition using the macro: form will usually
suffice. In practice such application is performed by the framework
when generating target code.

The forms residing in the code body of a macro: form have a
one-to-one correspondence to concatenative macros. Literals
found in the body of a macro: are mapped to compilation state
transformers that append qw instructions to the current code
list. Identifiers are mapped to function values using the
macro form, which fishes them out of the (macro)
dictionary.

Note that the objects produced by (macro+) and (macro:+) are different, although they have the same behaviour. The former
is a variable reference while the latter creates a new abstraction.
This is similar to the distinction between the Scheme expressions
+ and (lambda(ab)(+ab)).

Composing macros that are not in the (macro) dictionary is
possible using the unquote operation. The macro:
form behaves similar to quasiquote. Unquoted objects come
from the Scheme lexical environment and are interpreted as macros.

The first two sub-forms in a compositions form indicate the
target dictionary and body compiler respectively. The rest of the
body consists of a list of lists, where the first element of each list
is an identifier to which a macro will be associated in the
dictionary, and the rest of the list is a code body that’s passed to
the body compiler form.

1.2.2Primitives

The previous section describes how to compose existing code to create
new code by concatenation, and how to evaluate code into a form that
can be passed to the assembler. This section will describe how to
define primitive macros operating on stacks of target machine
instructions.

Creating an instance of a machine code instruction is done using the
op: form. It is exactly these objects that are produced when
concatenative macros are evaluated.

Real opcodes can be passed to the assembler to produce binary output.
Pseudo instructions cannot.

> (op-applyins10)

(9258)

> (op-applyins20)

(3963)

> (op-applyins30)

asm-pseudo-op: qw

However, pseudo instructions can be used to hold intermediate data
during the compilation phase. The following will illustrate the use
of a form to define new primitive operations. We’ll create a macro
add that will behave like the macro + encountered
before.

Creating new primtive macros is done with the patterns form.
Its first subform specifies the dictionary to which definitions are
associated. The rest of the forms contains lists of pattern and
template pairs. The following example defines the add macro
as not taking any input from the code stack, but producing an
addwf instruction as output.

(patterns(macro)

((add)([addwfPOSTDEC000])))

The templates in a pattern form are lists of forms that are
passed on to the op: form, resulting in lists of instruction
objects. Verifying if it works gives:

> (print-code(macro:add))

[addwf POSTDEC0 0 0]

This is an example of a non-optimizing macro which only performs code
generation. To add different behaviour for different input patterns,
extra clauses can be added.

(patterns(macro)

(([qwa]add)([addlwa]))

((add)([addwfPOSTDEC000])))

> (print-code(macro:123add))

[addlw 123]

> (print-code(macro:add))

[addwf POSTDEC0 0 0]

When a qw instruction appears in the input, it is
deconstructed and its operand is used as the operand of a
addlw operation. The patterns form is built on the
PLT Scheme match form, deconstructing a stack of
instructions according to input patterns, and constructing lists of
instructions to be added to the compilation state to replace the
matching top of stack.

Upto now our add doesn’t really perform compile-time
computation other than selecting a different instruction based on the
presence of literal data. Using the tv: form we can add
proper computation when there are two literals available. For now,
think of tv: as an RPN calculator behaving as in:

The tv: form will install wrappers to enable computations
that can only be performed after the assembler has assigned numerical
addresses to code labels. The use of this compile-time calculator
then leads to an implementation of the + macro whith 3
evaluation modes:

1.3More

Staapl contains a generic concatenative language parser in
staapl/rpn which is used to implement the languages
macro:, tv:, scat: and target:.
This language can be extended with prefix parsers to implement
a Forth-style prefix syntax for defining words.

At the core of the Forth code generator is an incremental compiler
which constructs a control flow graph as its first output pass.
Subsequent optimization passes operate on this structure.

Assembler opcodes can be specified using the instruction-set
special form defined in staapl/asm. This is used to define
an assembler for the PIC18 instruction set.

The distribution contains some example Forth code and library routines
that implement host to target tethering.

There is a significant body of code to perform run-time target access
through the target: language, and incremental code upload.
Staapl can emulate a standard Forth console.

Finally, Staapl provides the staaplc command line application
which can be used to compile stand-alone Forth programs to binary, for
upload with a microcontroller programmer tool.