crawler.frontier_num_top_level_dirs (collection.cfg setting)

Description

This optional parameter can be used to specify the number of top level directories to use when storing files that are required for a disk based frontier.

Default value

(empty)

Examples

If you are performing a very large crawl you may run into filesystem limits on the number of files allowed in a single directory. To get around this you might split the frontier files among 100 top level directories by setting: