The moderator nattered:>[I know there are Unicode versions of lex, such as the one from plan>9. And yes, you only need to store valid transitions. One technique>is to store the highest and lowest valid tokens in each state and a>vector of transitions [lowest,highest]. -John]

I seem to recall from the 9fans list (comp.os.plan9) that the Plan 9
lex actually doesn't do Unicode; noone at Bell Labs bothered to update it.
They seem to prefer hand-written scanners.