t.w.x.x.Scanner : class documentation

The Yapps scanner can work in context sensitive or context insensitive
modes. The token(i) method is used to retrieve the i-th token. It takes a
restrict set that limits the set of tokens it is allowed to return. In
context sensitive mode, this restrict set guides the scanner. In context
insensitive mode, there is no restriction (the set is always the full set
of tokens).

[token, ...] or None; if restrict is None,
then any token is allowed. You may call token(i) more than once. However,
the restrict set may never be larger than what was passed in on the first
call to token(i).