Package org.netbeans.api.lexer Description

The entrance point into Lexer API is
TokenHierarchy class with
its static methods that provide its instance for the given input source.
TokenHierarchy.tokenSequence()
allows to walk through the tokens that were created for the given input.
Token models a single token recognized in the input.
It carries a token identification TokenId returned by
Token.id()
and token text represented as CharSequence
returned by Token.text().
Language represents
all the token ids for a single language.
Each token id may belong to one or more token categories that allow to better operate
tokens of the same type (e.g. keywords or operators).