According to Gabriel Fenteany:
> Hi. I am indexing a large number of different servers. Some of the URLs
> point to http://foo.com/ but apparently the index file is not index.html but
> index.htm or default.html. Will htdig dig a site right if http://foo.com/> uses "index.htm" and not "index.html" If it would NOT dig the site with
> the the more standard index filename, what is the switch I'd use in the
> htdig.conf Point is, I don't want to have to check what the name of the
> entry page of all these kinds of sites are.
>
> I indexed a big list of sites, and most come up...but so far of the ones
> I've checked, only the ones that deviate from "index.html" are not showing
> up when the URL I have for them is http://foo.com/

As Torsten said, a lot of this depends on how the HTTP servers are set up.
In Apache, you can set the DirectoryIndex parameter in srm.conf, to
indicate which files are valid as directory indexes. E.g.:

But if you're digging multiple sites, you only want to remove the names
that are allowed as DirectoryIndex on ALL of the sites you dig, i.e. the
intersection of all the sets. Otherwise you may end up stripping off
names that aren't really directory indexes on some of the sites.

Finally, if you use the local_urls attribute, you should set the
local_default_doc attribute to the one name that is most commonly used for
directory indexes on the local file system. For any local directories
that don't have this file, htdig will fall back to the HTTP server to
get the directory index.