Frontera is a web crawling tool box, allowing to build crawlers of any scale and purpose. It includes:

crawl frontier framework managing when and what to crawl and checking for crawling goal* accomplishment,

workers, Scrapy wrappers, and data bus components to scale and distribute the crawler.

Frontera contain components to allow creation of fully-operational web crawler with Scrapy. Even though it was
originally designed for Scrapy, it can also be used with any other crawling framework/system.