Makes me wonder if the MySQL setup provided by pair is running LinuxThreads.

Clarification: I have no opinions or objections to the hardware/OS PM runs on. I was just geniunly curious how our MySQL server was compiled. Based on tye's reply, it looks like they'll be testing out hte LinuxThreads option soon.

Actually, PerlMonks runs on three FreeBSD boxes. We have one DB server and two web servers.

We had a lot of problems when the site was first moved from Linux to FreeBSD. It became clear (to me, anyway) that with our build of MySQL on FreeBSD (and this was back before the problem linked from MySQL on FreeBSD had been identified), a single long-running query could cause lots of other queries to block and that this was causing the site to completely lock up for several seconds or minutes (for example, whenever super search was used).

This problem was worked on at several fronts. vroom rewrote simple title search and super search to use full-text indexing, then I rewrote them both again. I remember reworking the backups repeatedly to get them to not lock up the site and reworking other maintenance tasks and writing breathe to get them to not lock up the site.

We all researched and eventually found the first reports of the problem and then possible fixes. ehdonhon rebuilt MySQL as suggested but the site just refused to stay up on it. We investigated switching back to Linux but never managed to get the resources needed for that.

I patched and recompiled Term::ReadKey so that it no longer was a CPU hog on FreeBSD so that we could use 'mytop' more often to monitor the DB. Then I reworked it and renamed it as sqltop and taught it how to run as a daemon, logging long-running queries and even killing ones that ran *too* long.

We finally had the site running pretty smoothly (though not exceedingly quickly).

Then pair.com gave the DB server a huge upgrade (I think the CPU speed quadrupled) and the site was pretty happy for a while... though the traffic volume continued to grow...

Last January the site was the busiest it had ever been. Traffic actually fell off a bit after that. This January we hit another peak and finally the web servers were having a hard time keeping up.

Luckily I'd been working on finding problems and fixing them so I had several patches to apply that would eventually help fix the problem and really help quantify how well we were fixing things.

Unfortunately, trying to deploy these was increasingly difficult because the web servers were often overloaded. I even ended up backing out a big set of behind-the-scenes changes even though my testing showed they increased web server load by less than 5%, because the site became nearly unusable for part of the next day.

And we started getting the occassional web server needing to be rebooted. So I was trying to put together some data to back me up so I could ask Pair for a third web server without feeling like I was just whining... when Pair said the web servers were due hardware upgrades. Yeah!

The web servers were certainly getting bogged down some of the time (you could easily notice this when logged into them). The load averages were higher than I expected (but usually not *really* high), but in any given 5-second period the web servers usually had some spare CPU cycles (usually quite a few). And they were only needing a fraction of their memory. My rather wild guess was that the web servers were alternating between everything waiting while MySQL got locked up for just a second or less and then lots of queries completing at once and being CPU starved to process them all.

In any case, today's doubling of the web server's CPUs from 1GHz to 2GHz is very welcome.

Fairly soon we should be testing MySQL v4 w/ LinuxThreads which should allow us to make better use of the DB server's power. And I'll be deploying efficiency improvements for quite a while. And demerphq just did a ton of work to make the best/worst nodes lists much more efficient (and other improvements). So the site might be quite happy for another year of growth in popularity.

Anyway, I wanted fill in a bit about the site's performance history. I hope that was informative.

I patched and recompiled Term::ReadKey so that it no longer was a CPU hog on FreeBSD so that we could use 'mytop' more often to monitor the DB. Then I reworked it and renamed it as sqltop and taught it how to run as a daemon, logging long-running queries and even killing ones that ran *too* long.

Hi, tye -- Unfortunately we couldn't make head or tail of your chatbox tip, "I just disabled the "compatability test", tim (FreeBSD is certainly compatible even though it fails that test)".

We're running 5.005_03 built for i386-freebsd on FreeBSD 4.9, and mytop-1.4 from freebsd ports. We tried ReadKey-2.21 and ReadKey-2.30, but runing mytop uses loads of system cpu:

Google found this post, but hasn't given us any clues to the patch. If you'd be willing to share it, and allow us (on your behalf, of course) to add it to a patch submitted to update /usr/ports/devel/p5-Term-ReadKey from 2.21, we'd really appreciate it.

I've you'd like to contact me via email I can be reached at tim.hunter atsign eircom dot net. Thanks!