Thunderstone Search Appliance Manual

This allows you to specify the URL of a plain text file containing a
list of site URLs to walk. This is an additional way of specifying
more Base URLs 3.4.3. This URL will be refetched each time
a rewalk is started. In the file, the list of URLs can be one URL per line (preferred) or delimited by any number of spaces.

Warning: Due to the nature of Stay Under, a large number of URL
URLs (1000+) in different directories will cause the walk to progress
very slowly, as all URLs encountered will need to be checked against
every one of those directories. In such a situation, we recommend
turning off Stay Under and instead writing your own
Required Prefix/Required REX expressions, which will be
more efficient.