A basic TokenStream.Tokenizer implementation that ignores whitespace but includes tokens for individual symbols, the period
('.'), single-quoted strings, double-quoted strings, whitespace-delimited words, and optionally comments.

Package org.modeshape.common.text Description

A set of utilities for working with text. Included is an Inflector class that transforms (English) works into singular,
plural, and human-readable forms, and is capable of transforming between camel-case, whitespace-delimited, or underscore-delimited
forms. There is also components that can encode and decode text.
Finally, this package defines a simple framework for tokenizing text (e.g., from files or stream)
making it very easy to create very straightforward parsers.