We appreciate the enormous support that our ABestWeb community has experienced over the many years it has served its members and sponsors. We have decided to exit this business and have placed the property up for sale and we are actively entertaining interest.

In the meantime, community members will be able to read but not post to ABestWeb beginning on Jan. 18.

We want to thank you for your numerous contributions and your ongoing support. If you have any questions, please let us know.

I got AV scooter practically living on my server for the last week or so.
Its getting hungrier by the minute.

I have a dedicated server with about 8 websites running on it now and AV is taking about 50 to 60% of the load. (lot a rewrites going on)

I tried to ban it (disalow) but i'm not sure if I miss a opportunity then.
Up to this day I never seen any traffic coming from AV (from the SERPs I mean)
OTOH it could bring some traffic after they completed the crawl?

What would you do?
Ban and save bandwith and server load, or let it get some because it could bring me some decent traffic?

I'm disallowing Scooter in all but one directory. It's not allowed anywhere near my rewrite stuff. I did this 2 months ago when it ate about 3000 pages and indexed roughly 1% of them.

I've seen one, count them, ONE, referrer from AV this month. That's up from zero, last month.

I hate to ban any legitimate spiders, but this one is a little too hungry, especially when few people actually use AV.

Inktomi is another hungry bugger. Their bots are only allowed in certain directorys. It's YahooSeeker is a pig the way it eats, it has more access than Scooter, but then, I get referrers from Yahoo, almost as many as Google.

Something to also keep in mind...Altavista is owned by Yahoo (acquired through it's overture purchase).

You never know what may develop next year. Yahoo may even merge the contents of the Inktomi/altavisa databases into one massively crawled database.

If it is unduly stressing your server, then by al means, disallow it. However, there may be other work-arounds you cna use to insure it only hits what you want it to hit.

I've also seen one of my larger (PR7) sites being hit hard by paractically all crawlers this week (scooter, ask jeeves/teoma, googlebot, and the infoamous MSN bot, which has chewed up about 20,000 pages in the last 3 days).

Personally, with the rapidly changing SE landscape (microsoft may even acquire google), I'm loathe to disallow any legtimate spider, for fear that it's index may get rooled into one of the big 3's databases.

I banned them on 2 sites, let them eat on the others, load is pretty descent now.
Just looked at the logs, scooter has gotten over 50K pages the last couple of day's, I think thats enough for now
If they show me some traffic for that I will remove the "deny".

One of the sites is hardly visited by anyone, bots or real people, so could make a nice studycase for the effect AV has.

BTW, I like to monitor with the combination:
TOP, for monitoring the complete server and
tail -f on specific domains.

A other thing I have seen a lot of suddenly is "FunWebProducts", anyone else noticed?

I'd be careful about banning a known good bot. As mentioned, you never know what might develop down the road. There must be a reason for all the spidering going on, it costs them money as well.

What I get are a few unidentified robots, and I've considered disallowing them, but I hate to do that since I'm not really sure what they're doing. If I discover they're up to no good, I'll disallow them immediately, but until then I'm going to just keep an eye on them.