Switch over to use token.c instead och abusing lex for tokenizing. This is a home-grown implementation (which is a quite straight-forward conversion of the scanner lex code) that is significantly faster (and smaller) than the lex code. There are still a bunch of places where more optimizations can be added though.