We are implementing htdig (v3.1.2 + the patch kit on Solaris 2.6) on our
main web server. One comment we have had is that there are alot of
duplicate search results pointing to the same web pages. This is usually
caused by having several different Unix symbolic links pointing to the
same directory/file in the web document tree.

Is there any way we can prevent the indexing of these duplicates? I see
from the mailing list archives that for previous versions of htdig there
were patches to fix this issue but they are not available for the current
version.

I see from the bug database the latest advice is to eliminate symbolic
links - however for many practical reasons it is not possible for us to
do this.

Is it for example possible to configure htdig to index our URLs via the
filesystem instead of HTTP (i.e using local_urls) and to ignore the
symbolic links?

How are people on the list working round this problem? Or is this an
unresolved bug I will need to (re)log with the htdig developers?