Hi:
As by the number of posts I have sent in you can tell I am experimenting
with Spidering and also AutoSwish. Thank you all for your help.
Here is a design flaw. I'm not knocking anyone as I think the software is
wonderful. I wish I knew "c" and Perl better to offer modifications.
I have a website that is 4+ years old. Back then we created a directory
(actually we have this problem in many directories) and instead of an
index.html we had a file named archives.html. We added ssi at some point
and since the search engines had the archive.html indexed we created
archives.shtml and turned the archive.html into a redirect page.
Later we created an index.html and inserted this code:
<META HTTP-EQUIV="Refresh" CONTENT="1;
URL=http://www.prophezine.com/search/database/archives.html">
Turns out that when I insert http://www.prophezine.com/search/database/ in
the config file it only indexes the index.html page that is returned. That
page has some meta tags but no body.
What is needed is a change to the spider to follow the refresh tag. I am
not sure of all the tags possible so there may be another to follow but this
should definitely be followed.
Thoughts?
Bob