Autopager is a Python package which detects and classifies pagination links.

License is MIT.

Installation

Install autopager with pip:

pip install autopager

Autopager depends on a few other packages like lxml and python-crfsuite;
it will try install them automatically, but you may need to consult
with installation docs for these packages if installation fails.

You also have to use AutoPager class if you’ve cloned repository from git;
prebuilt model is only available in pypi releases.

Detection Quality

Web pages can be very different; autopager tries to work for all websites,
but some errors are inevitable. As a very rough estimate, expect it to work
properly for 9/10 paginators on websites sampled from 1M international
most popular websites (according to Alexa Top).

Training Data

Data is stored at autopager/data. Raw HTML source code
is in autopager/data/html folder. Annotations are in autopager/data/data.csv
file; elements are stored as CSS selectors.

Training data is annotated with 5 non-empty classes:

PREV - previous page link

PAGE - a link to a specific page

NEXT - next page link

LAST - ‘got to last page’ link which is not just a number

FIRST - ‘got to first page’ link which is not just ‘1’ number

Because LAST and FIRST are relatively rare they are converted to PAGE
by pagination model. By using these classes during annotation it can be
possible to make model predict them as well in future, with more training
examples.

To add a new page to training data save it to an html file
and add a row to the data.csv file. It is helpful
to use http://selectorgadget.com/ extension to get CSS selectors.

Don’t worry if your CSS selectors don’t return <a> elements directly
(it is easy to occasionally select a parent or a child of an <a> element
when using SelectorGadget). If a selection itself is not <a> element
then parent <a> elements and children <a> elements are tried, this is
usually what is wanted because <a> tags are not nested on valid websites.

When using SelectorGadget pay special attention not to select anything other
than pagination elements. Always check element count displayed by
SelectorGadget and compare it to a number of elements you wanted to select.

Some websites change their DOM after rendering. This rarely affect paginator
elements, but sometimes it can happen. To prevent it instead of downloading
HTML file using “Save As..” browser menu option it is better to use
“Copy Outer HTML” in developer tools or render HTML using a headless browser
(e.g. Splash). If you do so make sure to put UTF-8 encoding to data.csv,
regardless of page encoding defined in HTTP headers or <meta> tags.