It's been particularly slow the last 48 hours. In some cases, the forum simply never loads.

I'm not going to complain about something not working when it is FREE to the community.

However, I vote for the KLOV model:

Make everyone donate $1 or $2 to post. Not only does this generate money to keep the site hosted, but it will also help keep the spammers out. In turn, it might make the mods life easier too. They would only have to approve accounts that have already donated. Instead of approving every new post that comes in.

I just saw a slowdown at about 3-4 mins past the hour, but it's come good again. It's just now a little past midnight US Eastern. In the next couple of hours if history is a guide, the site is going to slow again and then throw 500 errors.

Given the symptom (500 errors from a server running SMF) and the timing (daily, just after "midnight", in someone's US timezone at least) I'm willing to guess:

PHP is throwing "MySQL has gone away" errors into an appropriate log once calls out to the DB exceed its timeout

Web server is slow to respond while it still trusts MySQL might eventually have an answer for it, then starts sending quick 500s once it's sure it won't be back

Things I'd do

Check for active queries while the site is slow.Assuming there's direct access to the database (e.g. command line or even PHPMyAdmin), send this query to the site database as the site database's user:

SHOW FULL PROCESSLIST\G

If the database user has the necessary privileges this will braindump the active db queries. Things to look for are the lines with "Time:" and "State:" -- if there are heaps of rows and they're slow, well, yeah, database bottleneck. The queries themselves might give a clue exactly WHAT it's busy with, which might lead to some ideas about shifting that activity to some other time etc.

Check for any spikes in site traffic.

Count rows per minute in the web server's access.log file, look for any spiky patterns for IP addresses or User Agents. Could be as simple as something set to crawl the site at the same time each night.

Make sure someone with SMF knowledge is involved

I'm talking very generally here, a lot of the true fixes are going to require knowledge of how to tune SMF to avoid being wasteful. Perhaps there was a spamfest a few months ago that led to hundreds of thousands of dummy registrations in the database, now slowing down daily admin functions due to sheer size. Maybe a feature was enabled that runs some daily background processing and nobody remembered to turn it off. I don't know, I'm a Drupal guy, not an SMF guy, but all commodity web apps have their quirks and all can be abused from time to time

I'm gonna guess that the problem is in the SQL database. I had this issue when I was using velcom. I switched my server to Liquidweb and the problem went away. Also velcom did not update their php myadmin panel often enough and I was compromised by a mid eastern hacker.

You have to look at the benefits BYOAC has to offer, and what we have been receiving for free over the many years.

Logged

"I really do feel pity for those who go out of their way to perform a personal slight, than taking the effort in joining the conversation and creating some useful dialogue which would benefit the persons' perusal of this topic. Yet where would we be without the persistant antagonist?"

Migrating an application of this size and age off Simple Machines Forum to another application like Burning Board is likely a thousand times more work and risk and cost, than getting the hosting stabilised. Saint's running the latest stable version of SMF which demonstrates someone at least is doing the right thing and keeping it up to date.

I'd suggest passing the hat around is not a bad idea, with the goal of engaging the support required to diagnose, then repair or migrate the hosting depending on what's actually failing.

If your worried about moving to another platform, just do like I did and install XAMPP on your local drive then copy the entire forum and database and convert from there. I know there are tons of converter tools for PhPBB, vBulletin, SMF, UBB, IPB, MyBB, XenForo, etc, etc. SMF is nice because of the hook system when installing MODS, but it tends to be pretty taxing on the SQL. My personal preference would be PhPBB because it is so easy to work with. I've had no problem at all creating my own MOD's for it, even have a couple of them listed in the PhPBB MOD database. Not to mention there are tons of theme's available for it, and creating theme's is a breeze too.

One of the forums I owned had 500 plus sub forums (newsucks500.com now under new ownership and new platform), which, on most any other software, can really overheat those SQL database tables. PhPBB was able to handle it all just fine with no slowdown or crashing. Something I never would have attempted with SMF. In fact, the largest forum on the internet (gaiaonline.com), is nothing more then a heavily modified version of PhPBB.

Thank you for the offers of financial donations. It is within my means to be able to afford it if needed, so your intentions are gratefully acknowledged but respectfully declined

We're not on a hosted platform, I lease the entire server.

I've asked for assistance from someone with a level of expertise in the OS/config on the server. If this person is able to help then we'll be in good shape. If not, my intent is to lease a new server (this one is now about 3 years old) and migrate everything over to a fresh installation. I plan to stick with SMF unless it turns out the issue is SMF in some fashion. I don't believe it is, but obviously if I knew what it was it'd be resolved by now

It's going to take some more time. The person I've asked for help has their own life/work/family responsibilities, so time on fixing the server has to compete with that. If they aren't able to assist, in a couple of weeks I will start working 4x10 work weeks with 3 day weekends, and that will give me the extra time to purchase and start configuring the new server.

I appreciate everyone's patience and concern. If all else fails I will throw up the white flag and ask for additional help from the forum. For security's sake I am cautious about who I give access to the server directly, because it has your information on it in some fashion, but the option is on the table if needed.

I did finally figure out why my script to restart sql/web-server was failing, and as of this morning it automatically restarted things rather than having to wait for me to manually log on and restart them. This means when I sleep in on the weekend the forum won't be down until 9-10 AM

I just saw a slowdown at about 3-4 mins past the hour, but it's come good again. It's just now a little past midnight US Eastern. In the next couple of hours if history is a guide, the site is going to slow again and then throw 500 errors.

Given the symptom (500 errors from a server running SMF) and the timing (daily, just after "midnight", in someone's US timezone at least) I'm willing to guess:

PHP is throwing "MySQL has gone away" errors into an appropriate log once calls out to the DB exceed its timeout

Web server is slow to respond while it still trusts MySQL might eventually have an answer for it, then starts sending quick 500s once it's sure it won't be back

Things I'd do

Check for active queries while the site is slow.Assuming there's direct access to the database (e.g. command line or even PHPMyAdmin), send this query to the site database as the site database's user:

SHOW FULL PROCESSLIST\G

If the database user has the necessary privileges this will braindump the active db queries. Things to look for are the lines with "Time:" and "State:" -- if there are heaps of rows and they're slow, well, yeah, database bottleneck. The queries themselves might give a clue exactly WHAT it's busy with, which might lead to some ideas about shifting that activity to some other time etc.

Check for any spikes in site traffic.

Count rows per minute in the web server's access.log file, look for any spiky patterns for IP addresses or User Agents. Could be as simple as something set to crawl the site at the same time each night.

Make sure someone with SMF knowledge is involved

I'm talking very generally here, a lot of the true fixes are going to require knowledge of how to tune SMF to avoid being wasteful. Perhaps there was a spamfest a few months ago that led to hundreds of thousands of dummy registrations in the database, now slowing down daily admin functions due to sheer size. Maybe a feature was enabled that runs some daily background processing and nobody remembered to turn it off. I don't know, I'm a Drupal guy, not an SMF guy, but all commodity web apps have their quirks and all can be abused from time to time

Thank you Shock, I'll dig into these. While it's slow all day, it *is* consistent on when it finally gives up the ghost each day.

For any Linux (particularly Ubuntu) gurus out there, here is what a random sampling of iotop shows while the forum is running. The various bits that are showing up in the top change about 1-2 times per second. In a few minutes, I'll down the web server and mysql, and capture a few iotops while they aren't running for comparison. Unfortunately, I don't have a baseline to know what is and isn't normal.