I need your help! I am losing my mind trying to solve anotherbafflingmystery. For the past three or four months, I have been recording many404Errors generated from msnbot, Yahoo-Slurp, and other spider crawls. These errors result from invalid requests for URLs containing query strings such as the following:

https://perishablepress.com/press/page/2/?tag=spam

https://perishablepress.com/press/page/3/?tag=code

https://perishablepress.com/press/page/2/?tag=email

https://perishablepress.com/press/page/2/?tag=xhtml

https://perishablepress.com/press/page/4/?tag=notes

https://perishablepress.com/press/page/2/?tag=flash

https://perishablepress.com/press/page/2/?tag=links

https://perishablepress.com/press/page/3/?tag=theme

https://perishablepress.com/press/page/2/?tag=press

..plus hundreds and hundreds more 1. The URL pattern is always the same: a different page number followed by a query string containing one of the tags used here at Perishable Press, for example: “/?tag=something”. The problem is that there are no such links anywhere on the site. The site employs permalink format for all WordPress-generated links (e.g., post/page links, search queries, tag queries, etc.). In an effort to locate the source of these URLs, I have performed multiple, thorough searches through every aspect of my site (files, database, code, examples, etc.), and the results are always the same: nothing. UGH! What is the source of these misguided URLs? Hopefully, you will be able to help shed some light on this unexplained crawl behavior (please, I beg you!!).

More Information..

Many of these errors show up in the 404 log every day. They are almost always from either msnbot or Slurp 2 — I have never seen one from googlebot — but occasionally some other, unknown agent is involved. The requests happen in chronologically sporadic fashion; that is, relatively significant periods of time pass between each of the requests — they seldom occur in immediately successive fashion. Further, all of the queried tags are indeed used within the site, but they are all (as far as I know) referred to using permalink-formatted URLs. None of the tag archives are referred to via query string. In the past, however, all tags were referred to via query strings (and not permalinks). I forget the cause of the change, but this is something I plan to investigate further..

Anything and Everything..

Several weeks ago, I finally threw my hands up and decided to just block the spiders from requesting the query-based tag URLs to begin with. Supposedly, both Yahoo and MSNobeyrobots.txt directives, so I added the following, admittedly redundant rules:

Disallow: */?tag=*
Disallow: */*?*

As I said, that was several weeks ago, so the lack of positive results may have yet to manifest. In the meantime, while waiting for the spiders to check and follow the new robots.txt directives, here are a few of the other things that I have tried or considered:

using mod_rewrite to pattern match and redirect “tag” query strings to their permalink counterparts

Unfortunately, my efforts to solve this mystery have yielded zero fruits. What I need at this point are more leads to investigate. Hopefully I have explained the issue well enough to get your wheels turning, but if something isn’t clear, let me know. Otherwise, if you have any ideas or clues at all, please leave a comment or contact me directly. Thanks!

Check it out

Comments

I’m guessing they are taking information they’ve learned from some other site and trying to use it to navigate here, even though it fails every time and is totally wrong. The spiders don’t leave referrer information, do they?

I see a lot of strange things on my site… MSNBot is ever-present but never doing something it’s actually allowed to do.

Google Adsense bot actually follows users to every page they visit, especially when the links are visible only to registered users (which mediapartners google is NOT right now)

Strangely, even though the bot has no way to see onto the hidden pages, the ads are still relevant…

Yes, that seems to be the best explanation.. Awhile ago, I discovered some suspicious behavior where the Slurp crawler seemed to be using keywords from a few of my other sites (on the same server). The keywords were appended to existing URLs for this site, and resulted in a slew of 404 errors. This behavior is either an error or deliberate — each disconcerting for their own reasons. And, don’t even get me started about Yahoo disobeying robots.txt rules..

Hi yli, if Yahoo obeyed robots.txt directives, then yes, that would work. Unfortunately, Yahoo has shown a consistent pattern of disobeying robots.txt directives.

I have actually considered using something like htaccess to block Yahoo Slurp, but Yahoo continues to send modest amounts of traffic. If that ever changes, they’re blacklisted.

But there is no need to block Yahoo. Instead, we can block access to those random, nonexistent directories. Here is how to use htaccess to block access to those directories for any request (i.e., from any user agent):

I know I’m a little late to the party but Google WMT might shed some light on this if you can see whether any other sites are linking to a tag URL that doesn’t exist and getting the bots confused. Considering they are finding all sorts of wrongly formatted URL’s perhaps it was a sitemap issue?

Projects

About the site

Perishable Press is the work of Jeff Starr, professional developer, designer, author, and publisher with over 10 years of experience.
Check out some of Jeff's books and projects, follow on Twitter, or learn more »

Fun fact: Perishable Press has been online since 2005, and features over 800 articles and more than 11,000 comments. More stats »