Char Delimiters Separator

The char_delimiters_separator class is an implementation of the TokenizerFunction concept that can be used to
break text up into tokens. It is the default TokenizerFunction for
tokenizer and token_iterator_generator. An example is below.

Whether or not to return the delimiters that have been found. Note
that not all delimiters can be returned. See the other two parameters
for explanation.

returnable

This specifies the returnable delimiters. These are the delimiters
that can be returned as tokens when return_delims is true. Since these
are typically punctuation, if a 0 is provided as the argument, then the
returnable delmiters will be all characters Cfor which std::ispunct(C)
yields a true value. If an argument of "" is provided, then this is
taken to mean that there are noreturnable delimiters.

nonreturnable

This specifies the nonreturnable delimiters. These are delimiters
that cannot be returned as tokens. Since these are typically
whitespace, if 0 is specified as an argument, then the nonreturnable
delimiters will be all characters C for which std::isspace(C) yields a
true value. If an argument of "" is provided, then this is taken to
mean that there are no non-returnable delimiters.

The reason there is a distinction between nonreturnable and returnable
delimiters is that some delimiters are just used to split up tokens and are
nothing more. Take for example the following string "b c +". Assume you are
writing a simple calculator to parse expression in post fix notation. While
both the space and the + separate tokens, you only only interested in the +
and not in the space. Indeed having the space returned as a token would
only complicate your code. In this case you would specify + as a
returnable, and space as a nonreturnable delimiter.

To use this class, pass an object of it anywhere a TokenizerFunction
object is required.