The lexer will either analyze the tokens identifying the
type or collect raw input until a terminating delimiter
is found.

Package net.sf.eBus.text Description

Lexer takes a given input and
returns a series of analyzed tokens. These tokens contain:

A token type. May be either a pre-defined or
user-defined.

The token text value as collected from the input.

The Java object created from the token text.

The line number on which the token appears.

The most important difference between Lexer and
java.util.StringTokenizer is that Lexer analyzes the
tokens into types. Further, tokens are not created by
breaking a line based on inter-field separator characters or
by regular expression. Instead, Lexer searches for tokens
based on types. This lexical package is designed to work with
a higher level parser rather than a line-by-line text
processor.