NFAScanner

Why

We've all written custom loaders for misc text formats a dozen times or more. A compact, simple, generic, reusable system would be nice. I'd also like to be able to configure it at run-time, so I don't have to recompile if I change one or more tokens, and of course, working on compilers and related tech is just good fun.

What

A lexical analyzer that is simple, flexible, and easy to use. Tokens are defined via x-macros. These are used to configure an instance of NFAScanner at run-time, generating the required state machines. Feed your source text in, and get your tokens out. Some rudimentary error information is present, giving you an idea of where any unmatchable token was encountered.

Here's are a few of the state machine definitions that you might require for a C-like programming language…

// TokenDecl.h - definition of some tokens for a C-like programming language.