org.modeshape.sequencer.ddl
Class DdlTokenStream

A TokenStream implementation designed around requirements for tokenizing and parsing DDL statements.

Because of the complexity of DDL, it was necessary to extend TokenStream in order to override the basic tokenizer to
tokenize the in-line comments prefixed with "--". In addition, because there is not a default ddl command (or statement)
terminator, an override method was added to TokenStream to allow re-tokenizing the initial tokens to re-type the
tokens, remove tokens, or any other operation to simplify parsing.

In this case, both reserved words (or key words) and statement start phrases can be registered prior to the TokenStream
's start() method. Any resulting tokens that match the registered string values will be re-typed to identify them as key words
(DdlTokenizer.KEYWORD) or statement start phrases (DdlTokenizer.STATEMENT_KEY).