Destructor

Member Functions

Adds a token class to the tokenizer. The tokenizer takes ownership of the token and deletes it when no longer needed. Comment and whitespace tokens will be marked as ignorable, which means that next() will not return them.

Adds a token class to the tokenizer. The tokenizer takes ownership of the token and deletes it when no longer needed. If ignore is true, the token will be marked as ignorable, which means that next() will not return it.

Extracts the next token from the input stream. Returns a pointer to an EOFToken if there are no more characters to read. Returns a pointer to an InvalidToken if an invalid character is encountered. If a token is marked as ignorable, it will not be returned, and the next token will be examined. Never returns a NULL pointer. You must not delete the token returned by next().