Token filter

This callback is called whenever TeX needs a new token. No arguments are passed to the callback, and the return value should be either a Lua table representing a to-be-processed token, a table consisting of a list of such tokens, or something else like nil or an empty table. Since no argument is passed, if you want to get the next token in the document, you should use the token.get_next() function. Your lua function should therefore look like this:

function()
return <table> token
end

If your Lua function does not return a table representing a valid token, it will be immediately called again, until it eventually does return a useful token or tokenlist (or until you reset the callback value to nil). If your function returns a single usable token, then that token will be processed by LuaTeX immediately. If the function returns a token list (a table consisting of a list of consecutive token tables), then that list will be pushed to the input stack at a completely new token list level. If what is passed to TeX is expandable, then the result of expansion is inserted into input and token.get_next() will grab it.

Note. This callback will be eventually removed and replaced by a new token library. See this message on the luatex list.

Note this is not strictly equivalent to the real XeTeX thing, because the latter operates on the character lever and as implemented a macro breaks the char “chain”. So, a\AA a, with \def\AA{A}, doesn't work as expected.