Server issues: an update

I'm trying to keep the creaking VPS alive until I'm in a financial state where I can upgrade. (Still have to sell my house in suburban Cleveland, unfortunately.)

Cyburbia was getting hammered by a South Korean search engine bot. I banned the IP addresses, and the server seems to be operating much smoother; still on the slow side, but not as bad. You'll also probably notice that the "Users Online" count will be much lower than usual.

Thanks for your patience these past few months. You've all been great!

Growth for growth's sake is the ideology of the cancer cell. -- Edward Abbey

For some reason, Cyburbia seems to be the only site getting absolutely hammered with spider bots from Daum. (There's no other reports of aggressive Daum activity online, or at least none that I could find.) Daum is supposedly a legitimate Korean search engine. They're been hitting Cyburbia every half second to five seconds, and they ignore robots.txt, a file that contains instructions for search engines that visit the site. In the past week, Daum bots account for about 1/4 to 1/3 of all traffic on Cyburbia.

I tried to drop Daum IP blocks in iptables, but unfortunately for some reason iptables on my server is flushed when the server is restarted. Daumoa/2.0 bots are hitting the Cyburbia server so hard I think it's been a major factor in the recent, frequent server crashes.

I think that's all of them. When I tried banning a range early on, the bots would move to a different IP block.

I resorted to blocking all of these ranges in .htaccess. This means results for Cyburbia will drop in South Korea's leading search engine. However, Cyburbia does not rank very highly in searches for "urban planning" in Daum, and there have been very few active users from South Korea on Cyburbia through the years.

Also, there was a line in .htaccess that somehow caused the web server to perform domain name lookups on every visitor to the site, despite instructions in the server configuration not to do so. This causes the server to slow down. The errant line in .htaccess was removed, so there will no longer be domain name lookups.

I added a script that will check every minute to see if Apache (the Web server program) is running. If it is not, it will restart Apache automatically.

Growth for growth's sake is the ideology of the cancer cell. -- Edward Abbey

EDIT: It looks like the server can either be fast and efficient with memory and load, but crash-prone, or slow and inefficient, but more reliable. I'm trying settings for the latter, and seeing how that works.

In the coming weeks, Cyburbia should be moving to a new VPS at the same host. We desperately have to upgrade to MySQL 5, but that means the Drupal integration -- the front page, feeds, and so on -- will break. For a while after the move, Cyburbia will consist of just the forums and gallery, until I can reintegrate the site with Drupal and re-enter, by hand, all the articles and blog/RSS feed information.

Growth for growth's sake is the ideology of the cancer cell. -- Edward Abbey

I freely admit, I'm not that versed in behind-the-scenes stuff on computers. I'm the typical end-user. I want to turn the dang thing on, and it should work. When it doesn't, I call either Tech Support (at work) or Hubby (at home). So the details of what you're going through with these server problems just goes in one eye and out the other.

These problems sound like they are starting to get on your last nerve, and that's not good for you. Hopefully the upgrade/whatever else needs to happen, happens soon, for your sanity!

You take very good care of us here on Cyburbia, and I appreciate it.

Kudos, and Good Luck!

"We do not need any other Tutankhamun's tomb with all its treasures. We need context. We need understanding. We need knowledge of historical events to tie them together. We don't know much. Of course we know a lot, but it is context that's missing, not treasures." - Werner Herzog, in Archaeology, March/April 2011

Thanks Dan for a while I thought our IT folks were screwing with me as about every third attempt to access the site I get denied. Hang in there Dan we all super appreciate what you do and are very willing to open the pocketbook to help out again.

Remember this motto to live by: "Life should NOT be a journey to the grave with the intention of arriving safely in an attractive well preserved body, but rather to skid in sideways, chocolate in one hand, martini in the other, body thoroughly used up, totally worn out and screaming 'WOO- HOO what a ride!'"