CONSTRUCTORS

new

pattern - A string specifying a Perl-syntax regular expression which should match one token. The default value is \w+(?:[\x{2019}']\w+)*, which matches “it’s” as well as “it” and “O’Henry’s” as well as “Henry”.

METHODS

transform

my $inversion = $regex_tokenizer->transform($inversion);

Take a single Inversion as input and returns an Inversion, either the same one (presumably transformed in some way), or a new one.