Spiders Are Tricking My Session Management

In order to cut down on variables that are set on the server, I attempt to turn off session management for spiders so that no session variables need to be created. I do this based on user agents and black-listed IP addresses. However, recently, I have been getting a slew of hits from what I assume are spiders that have regular user agents:

Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)

Since I can't use that, I thought I would black list the IP addresses, but it seems that the spider is sending a randomized remote address for each page request. The following IP addresses all came from some sort of crawler within two minutes:

I know that it was a crawler because they all had the same http referer, which was my home page and not all of the requested pages are available from the home page, which means the referer was being set manually. This is so irritating! Now, I have dozens upon dozens of sessions being created on the server that will last 20 minutes without being used twice. That is poor memory management.

Why is the spider doing this? I suppose this is to stop people from serving up different content based on spiders, but that is not my purpose. Having no session management does not server different content. It just turns off certain server-side tracking. Uggg.

I usually don't bother to set a session until the user is authenticated or actually did something worth while to track, I do start sessions if an error occurs and once an error does occur I refer to the current and referring URL as well as store all variables being posted or submitted. So that I can better understand WTF the user is doing to cause the error. ( It could be my fault... but I prefer to blame the user )

As for the user agent being the same that is a little bit odd. It could be websites that are taking screenshots of your page --> ref http://www.browsershots.org or something similar?

It could also very well be an attempt to take down your site over load it.

I know I'm making this longer than what it should be but a user from a local network eg.) school, office, etc... could be using software that changes the ip each time they go to a new page... I have used a similar program when I was in school... ( changing my grades xD )

but anyways to have a little bit more ease on your session management you could always create a cookie for the user with a #createUUID()# or whatever and also store it in a database with a few other fields eg.) Ip, UUID, expires, and then check to see if it exsist and is still valid if it exist then you can continue to use the same session or not make a new one. If it doesn't exist you can always check to see if the user has cookies enabled via javascript and pass it to your coldfusion [ if i'm not mistaking a user has to have cookies enabled for a session to properly work unless your passing them through the URL ]

Well I have said enough so far... I think I will leave it at that. If I have left anything out just say something...

I definitely like the idea of storing multiple values to compare against on subsequent page hits. I would, however, probably just store that in the COOKIE and the SESSION to compare against; that way I don't have to hit the database each time.

I am the co-founder and lead engineer at InVision App, Inc — the world's leading prototyping,
collaboration & workflow platform. I also rock out in JavaScript and ColdFusion 24x7 and I dream about
promise resolving asynchronously.