I am having the same problem. I am trying to log a few KB caches I found today but I am not allowed in. I sent geocache.com an email telling them I will not renew my premium account if this problem is not solved. I assume all he needs is more server space or more bandwidth. Am I right?

First, given that a person is seeking a certain dataset from the site, a bot will generate less total load than the same queries posed by the 'clicky-clicky-scroll-clicky-clicky' process. In the case of non-PQ queries, this is because the bots do not request any of the images on the site. In the case of PQ-queries, this is because with a bot there is no need to schedule daily PQ's at all, they can be easily performed only when you need them.

Second, bots are used because Jeremy has refused to provide features that people have repeatedly asked for. Among the things I can think of off the top of my head are:

2a) PQ's are limited to 500 caches, so you either have to manually generate multiple overlapping PQ's to get the entire metro area (increasing server load), or simulate GPX queries by scraping thru the web interface. This is exacerabated because Jeremy has provided only circular regions, not rectangular regions. Rectangular regions can be non-overlapping and do not require great-circle calculations. To accomplish the same with circular regions requires a 44% overlap and great circle calcs.

2b) PQ's are limited to 5 logs, which is often not enough. Once again, you are forced to use (either manually or with a bot) the web interface to see a sufficient number of logs.

2c) New cache notifications. If you want to be an FTF, you need timely new cache notifications. An email on Thursday does not cut the mustard. But AFAIK, gc.com does not offer instant email notofication when a new cache is placed. So people are forced to poll the site with a bot multiple times per day.

2d) Stats. Need I say more?

2e) Off-site cache listing and log backup.

2f) Search for all "latest logs" in an area so on a rainy day you can read up what your fellow cachers have been doing.

Third, Jeremy has made a number of changes in the last 9 months that make a bots job harder to do. The timing of these changes makes it pretty clear that he is aiming these changes at bots and not to provide new functionailty. Bots aren't going away until gc.com provides what people need from the site. So he has reaped what he has sown.

Fourth, Jeremy has not paid attention to the clicky-clicky usage pattern of humans accessing the site, and has not optimized the site for the access patterns that humans will generate. This generates additional and unneeded load, which is not caused by bots at all, that he could correct with a redesign of the site. Especially when in "logging caches mode", a large number of unnecessary database queries and page loads are generated.

Fifth, there was a fifth point, but I got brain block and can't remember it right now. I reserve the right to edit this post when my brain recovers.

Fifth (remembered now), the latest feature of modern web browsers is to pre-fetch pages in advance of the user requesting them. Mozilla does this, and I'm told that IE6 has something similar but not as good (I could be wrong about IE6). Its entirely possible that some of the increased load on the site is due to the growing popularity of this browser feature. Unfortunately, this browser feature looks and acts a lot like a bot, but without the precision targetting of just the data required that only a well-written bot can do.

Sixth, Jeremy is as guilty of leaching as the people he condemns. For example, when a new cache is placed, the map image comes from the US Census Tiger map server. New caches generate lots of website activity, and he is generating load on the Census bureaus server for his commercial purposes. The TMS people have requested that commercial sites do not do this. Hypocrit? You decide.

Jeremy wants to run the site "for profit". Unfortunately, this has forced him into a site design that does not scale very well other than by throwing hardware at the problem. He has made his bed and now he must lie in it. If the profits of the site are not sufficient to support the required new hardware investment every 3 months, then perhaps he should rethink his business. An open database would allow for a very scalable design, potentially harnessing the power of the computers of every geocacher.

So am I part of the problem? You decide.... I run a 'newest caches' bot every day at 5AM that scrapes 2 web pages. I run a 'stats' bot twice a week (Monday and Friday mornings). I run an FTF bot every hour that scrapes 1 web page. I don't use daily scheduled PQ's, rather, I run a bot-generated on-demand PQ 10 minutes before I leave the house. On average I run this about twice a week (some weeks never at alll, some weeks more than twice, depends on whether I am going out or not). I am a paying member, and I don't hunt MOCs or TBs. Every few months I use a bot to do an incremental backup of caches I have placed or found, including logs. This is always run during the week in the wee hours.

I'll probably end up getting banned from gc.com for posting this, but it had to be said.

In his gc.com forum posts on the same topic, it seems to me that Jeremy claims it's a bug in the system, and not related to the amount of users on the system. But, when I cache during the week, I'll go caching, then go to work at 3:30pm, and write my logs after I get home at 2:30am, and have no problems. Then when I cache on Saturday or Sunday, I'll write my logs that evening when everyone else does, and keep getting the Server to Busy thing. Last night, it took me over two hours to log 11 finds and 2 DNFs--good thing Saturday is a bad TV night and I didn't have any other plans. I'm also a Charter Member, why do I get chosen as the deadlock victim?_________________There comes a time in every young boy's life when he gets an irresistible urge to seek buried treasure.--Mark Twain

Yes, I see that Jeremy has now backed off from blaming the 'bots' for his troubles.

He even started a new thread with essentially the same subject line as the old thread. No doubt he's hoping the old thread will fall off the first page and he can save some face.

Now, the data is that an "astounding" 50,000 logs were created during the last week, and this may be the source of his troubles. Even if everybody decided to log those 50,000 logs/week in just 4 hours on Saturday evening and 4 hours on Sunday evening that would amount to a transaction rate of less than 2 logs per second for the servers to handle. This really is not a large load.

I am glad to see Jeremy focusing now on the real source of the problem, which is the architecture of his system and the programming behind it. Unfortunately, he seems to be applying bandaid fixes (bailing water from a boat with a teaspoon). I expect he will get things under control for the time being, only to have this problem reoccur later this year. Its an architectural issue that he's going to have to face eventually. I saw he was looking for resumes

I also saw that robertlipe gently thru down the gauntlet for Jeremy to open up the source code so that more eyeballs could work on the problem. Nice chess move Robert! They will find Bin Laden before Jeremy opens his code, though.

If we go a period of time with no trouble and then the trouble surfaces again, I bet he will blame the 'bots' first. Anybody wanna take the bet?

Even if everybody decided to log those 50,000 logs/week in just 4 hours on Saturday evening and 4 hours on Sunday evening that would amount to a transaction rate of less than 2 logs per second for the servers to handle.

I did the math on that, too. (Even if you throw in some skew for the earth's rotation, the habits of people preferring to sleep when it's dark, and the fact that over 23% of the worlds caches exist in one time zone, and it's still not a lot.) If the site was well designed, it should be able to run on a nice laptop. If you put it on a Real Computer, there's just no reason it should be huffing and puffing.

Quote:

I expect he will get things under control for the time being, only to have this problem reoccur later this year.

History has proven this to be true.

Quote:

If we go a period of time with no trouble and then the trouble surfaces again, I bet he will blame the 'bots' first. Anybody wanna take the bet?

History has proven that's a sucker bet. But it's amazing how much backlash there is in the forums every time it comes up against those evil page scrapers.

Oh, and history has proven that posts like mine and Rick's will be utterly ignored.

Oh, and history has proven that posts like mine and Rick's will be utterly ignored.

They actually defend him to the end of the earth. They keep ignoring the fact that he has already blamed the bots for causing this issue two weekends ago. HE LIED TO YOU AND YOU STILL SUPPORT HIM. They keep having excuses like "you make your own site", "he's now asking for resumes", "he never expected it to become this big", etc.

I have been pretty disappointed in the past but this was ridiculous. It's just the fact that his supporters somehow outweigh those that are against him (at least vocally... who knows how much work their forum moderators do against the cause though).

Thanx for the welcome. If me joining the forum messes up your local stats site, I'll gladly unsubscribe or ask to be excluded or whatever. In TN we already have a lovely stats site so I don't need to skew your bell curves. (As an aside, you guys should get a piece of that action.) Rick and I have been friends and were coworkers for many years. I probably helped bait him into geocaching.

You're exactly right, Silent Bob. The domination of 'fan boys' in that crowd often gives it a ring of the Ministry of Truth. There are lots of things spoken and repeated as gospel that real techies know hold no water at all.

Thanx for the welcome. If me joining the forum messes up your local stats site, I'll gladly unsubscribe or ask to be excluded or whatever.

Thanks for joining. You don't mess up anything. RickRich's script simply takes the userlist from these forums and finds their geocaching profile... so mtn-man, etc. are in our stats as well, while some Minnesota cachers that have never been to these forums are currently excluded... no big deal.

It's good to have more people around that are knowledgable on such topics.

I wouldn't exactly call it whining. I call it stating facts. I am less concerned about the issues the site has been having. I am quite concerned w/the fact that Jeremy blatantly lied about the cause of the problems and those that support him continue to do so while making poor excuses and blatant denial of the fact that he lied about the problems last weekend.

Well, I might be a guilty recipient for some cheese... but there are a lot of us who are taking action and doing a lot to improve and bring positive change to the technical side to geocaching. Because of Groundspeak and their policies, a lot of efforts have cropped up such as RickRich's scripts, Buxley's Waypoint, Navicache, Scout's stuff, ubercaching.com, Keyori, opencaching.org and other related efforts such as gpsvisualizer and gpsbabel to making geocaching more usable, accessable and in the end more fun for geocachers. Part of this is to create a geocaching world that's not dictacted by Jeremy Irish and his pocketbook. I wouldn't have anything against him if he listened to his users (which would in the end bring less burden on him and his servers), instead of listening on how he can capitalize on our free sport. I don't mind people making money - but I don't like it when it's at the expense of usability, and when they're liars as he demonstrated again recently.