org.modeshape.cnd
Class CndTokenizer

A TokenStream.Tokenizer implementation that adheres to the CND format by ignoring whitespace while including tokens for individual
symbols, the period ('.'), single-quoted strings, double-quoted strings, whitespace-delimited words, and optionally comments.
This tokenizer optionally includes comments and vendor extensions.

Field Summary

static int

COMMENT
The token type for tokens that consist of all the characters between "/*" and "*/" or between "//" and the next line
terminator (e.g., '\n', '\r' or "\r\n").

static int

DECIMAL
The token type for tokens that consist of an individual '.' character.

static int

DOUBLE_QUOTED_STRING
The token type for tokens that consist of all the characters within double-quotes.

static int

SINGLE_QUOTED_STRING
The token type for tokens that consist of all the characters within single-quotes.

static int

SYMBOL
The token type for tokens that consist of an individual "symbol" character.

static int

VENDOR_EXTENSION
The token type for the token containing a vendor extension block.

static int

WORD
The token type for tokens that represent an unquoted string containing a character sequence made up of non-whitespace and
non-symbol characters.