Macros are basically stored as raw text, not as token runs (as in GCC's libcpp for example). The body of simple #defines without parameters is stored as one string. Macros with parameters are stored as sequence of "macro tokens". There are three types of macro tokens:

text("<text>")

Raw text, but spaces and empty lines trimmed (like in a #define without parameters)

textw("<wstring text>")

Same as above, just for Unicode input.

parameter(index)

A macro parameter was used here in the declaration. The index specifies which one. During expansion, the text of argument(index) is inserted where the parameter was in the declaration.

stringify_parameter(index)

Same as above, except the argument will be stringified during expansion.

Note: macro tokens are actually symb.bi:FB_DEFTOK structures, and they contain an id field holding on of the FB_DEFTOK_TYPE_* values to tell what they contain.

Storing macros as text is a fairly easy implementation, but it requires to re-parse the macro body over and over again. For example, since GCC works with preprocessing tokens and tokenruns, macros are stored as tokens, making expansion very fast, because there is no need to tokenize the macro body again and again. fbc's implementation is not as flexible and maybe not as efficient, but is less complex (regarding code and memory management) and has an upside too: Implementation of ## (PP token merge) is trivial. ## simply is omitted while recording the macro's body, where as in token runs the tokens need to be merged explicitly.

When are macros expanded?

Because of token look ahead, macros must be expanded during tokenization, otherwise the wrong tokens might be loaded into the token queue. Afterall the parser should only get to see the final tokens, even during look ahead.

In lexNextToken(), each alphanumeric identifier is looked up in the symb module to check whether it is a keyword or a macro. Macros and keywords are kept in the same hash table. Note that macros cannot have the name of keywords; "#define integer" causes an error. If a macro is detected, it is immediately expanded, a process also called "loading" the macro (pp-define.bas:ppDefineLoad()).

Macro call parsing

If the macro takes arguments, the macro "call" must be parsed, much like a function call, syntax-wise. Since macro expansion already happens in lexNextToken(), the source of tokens, the parsing here is a little tricky. Forward movement is only possible by replacing (and losing) the current token. The token queue and token look ahead cannot be relied upon. Instead it can only replace the current token to move forward while parsing the macro's arguments.

Since lexNextToken() is used to parse the arguments, macros in the arguments themselves are recursively macro-expanded while the arguments are being parsed and recorded in text form. The argument texts are stored for use during the expansion.

So, a macro's arguments are expanded before that macro itself is expanded, which could be seen as both good and bad feature:

#define stringify(s) #s
stringify(__LINE__)

results in 2 in FB, but __LINE__ in C, because in C, macro parameters are not expanded when used with # or ##. In C, two macros have to be used to get the 2:

The expansion text is a string build up from the macro's body tokens. For macro parameters, the argument text is retrieved from the argument array created by the macro call parser, using the indices stored in the parameter tokens. Parameter stringification is done here.

There is a specialty for the builtin defines (__LINE__, __FUNCTION__, __FB_DEBUG__, etc.):
A callback is used to retrieve their "value". For example: __LINE__'s callback simply returns a string containing the lexer's current line number.

Expansion

The macro expansion text (deftext) is stored by the lexer, and now it will read characters from there for a while, instead of reading from the file input buffer. Skipping chars in the macro text is like skipping chars in the file input: Once skipped it's lost, there is no going back. So, there never is "old" (parsed) macro text, only the current char and to-be-parsed text. New macro text is prepended to the front of existing macro text. That way macros inside macros are expanded.

This implementation does not (easily) allow to detect macro recursion. It would be hard to keep track of which characters in the macro text buffer belong to which macro, but that would be needed to be able to push and pop macros properly. It could be done more easily with a token run implementation as seen in GCC's libcpp. However C doesn't allow recursive macros in the first place: In C, a macro's identifier is undefined (does not trigger expansion) inside that macro's body. That is not the case in fbc, because (again) a way to detect when a macro body ends is not implemented.

Currently fbc only keeps track of the first (toplevel) macro expanded, because it's easy to detect when that specific macro's end is reached: as soon as there is no more macro text.