Project description

Text segmentation is the task of splitting up any amount of text into segments by placing boundaries between some atomic unit (e.g., morphemes, words, lines, sentences, paragraphs, sections, etc.). It’s a common pre-processing step in many Natural Language Processing (NLP) tasks.

This package is a collection of metrics and for comparing text segmentations and evaluating automatic text segmenters. Both new (Boundary Similarity, Segmentation Similarity) and traditional (WindowDiff, Pk) are included, as well as inter-coder agreement coefficients and confusion matrices based upon a boundary edit distance.