Geoff Hutchison wrote:
>
> >2) Does Htdig support "Stop Words", or as I understand: When Htdig hits a
> >word in a stop
> >word list it will stop indexing the individual page.
>
> Not to that extent. I've mostly heard the terms "stop words" to refer
> to words that should be ignored from searches or indexing. It does
> this. See <http://www.htdig.org/attrs.html#bad_words_list>

This is quite common misunderstanding of the term "stop word" ;-)

However, page indexing can be controlled by either robots exclusion
standards
(i.e. "robots" meta tag) or <--htdig_noindex--> ... <--/htdig_noindex-->
comments inside of documents. The latter is probably what Brett is
looking
for.