Create a String::Tokeniser, tokenises $sentence and resets the token counter.

The next argument determines how a ``token'' is defined: a value of 0 or undef determines that underscores are included in a token; -1 states that they are not. Alternatively, you can supply your own regular expression which will be fed to a split to determine the tokens.

Then may optionally follow a list of exceptions: tokens that would be split in two, but should be treated as one.

Assuming a string of tokens will end in $string, returns everything from the current `pointer' position until the string is found. Returns a two-element list: firsly, why the search terminated, (either EOF meaning we hit the end of the token list without success, or FOUND meaning $string was found.) and the rest of the tokens upto and including $string (or the end of the list, whichever was soonest).