IndicTokenizer

Method Detail

isTokenChar

Returns true iff a codepoint should be included in a token. This tokenizer
generates as tokens adjacent sequences of codepoints which satisfy this
predicate. Codepoints for which this is false are used to define token
boundaries and are not included in tokens.