I’m worried about the sigterm termination and the max hosts error every time MySQL starts I will be switching hardware soon and if these are underlying database issues I would like to resolve them before migration.

at least I’m not seeing the locked delete entries any more. I think I have a old deleted snapin associated with some hosts what would be a command to select and delete it from my sql it may be causing the apache errors I am getting now.

Can you elaborate on that? Will it try to come alive later on? If the DB connection is broke, can fog try to restart mysql? Can there be a limit on how often it gets restarted? Sounds like it’s living dangerously.

I’m not 100% sure how much I like approaching things in this manner, but i’ve now made it so FOG will die if the db connection cannot happen. This should stop spamming logs with “mysqli::query() couldn’t fetch mysqli” error messages. It also “deadens” the page right then and there. This way resources aren’t trying to get data as in the “main page” as used to be the case.

This isn’t going to fix the issue here, but should help in lessening errors being thrown all over the place.

public function cleanInvalidEntries() {
// $this->DB->query("DELETE FROM aloLog WHERE alID=0");
// $this->DB->query("DELETE FROM clientUpdates WHERE cuID=0");
// $this->DB->query("DELETE FROM dirCleaner WHERE dcID=0");
// $this->DB->query("DELETE FROM globalSettings WHERE settingID=0");
// $this->DB->query("DELETE FROM greenFog WHERE gfID=0");
// $this->DB->query("DELETE FROM groupMembers WHERE gmID=0 OR gmHostID=0 OR gmGroupID=0");
// $this->DB->query("DELETE FROM groups WHERE groupID=0");
// $this->DB->query("DELETE FROM history WHERE hID=0");
// $this->DB->query("DELETE FROM hostAutoLogOut WHERE haloID=0 OR haloHostID=0");
// $this->DB->query("DELETE FROM hostMAC where hmID=0 OR hmHostID=0");
// $this->DB->query("DELETE FROM hosts WHERE hostID=0");
// $this->DB->query("DELETE FROM hostScreenSettings WHERE hssID=0 OR hssHostID=0");
// $this->DB->query("DELETE FROM imageGroupAssoc WHERE igaID=0 OR igaImageID=0 OR igaStorageGroupID=0");
// $this->DB->query("DELETE FROM imagePartitionTypes WHERE imagePartitionTypeID=0");
// $this->DB->query("DELETE FROM images WHERE imageID=0");
// $this->DB->query("DELETE FROM imageTypes WHERE imageTypeID=0");
// $this->DB->query("DELETE FROM imagingLog WHERE ilID=0 OR ilHostID=0");
// $this->DB->query("DELETE FROM inventory WHERE iID=0 OR iHostID=0");
// $this->DB->query("DELETE FROM ipxeTable WHERE ipxeID=0");
// $this->DB->query("DELETE FROM keySequence WHERE ksID=0");
// $this->DB->query("DELETE FROM modules WHERE id=0");
// $this->DB->query("DELETE FROM moduleStatusByHost WHERE msID=0 OR msHostID=0 OR msModuleID=0");
// $this->DB->query("DELETE FROM multicastSessions WHERE msID=0 OR msNFSGroupID=0");
// $this->DB->query("DELETE FROM multicastSessionsAssoc WHERE msaID=0 OR msID=0 OR tID=0");
// $this->DB->query("DELETE FROM nfsFailures WHERE nfID=0 OR nfNodeID=0 OR nfTaskID=0 OR nfHostID=0 OR nfGroupID=0");
// $this->DB->query("DELETE FROM nfsGroupMembers WHERE ngmID=0 OR ngmGroupID=0");
// $this->DB->query("DELETE FROM nfsGroups WHERE ngID=0");
// $this->DB->query("DELETE FROM os WHERE osID=0");
// $this->DB->query("DELETE FROM oui WHERE ouiID=0");
// $this->DB->query("DELETE FROM peer WHERE id=0");
// $this->DB->query("DELETE FROM peer_torrent WHERE id=0 OR peer_id=0 OR torrent_id=0");
// $this->DB->query("DELETE FROM plugins WHERE pID=0");
// $this->DB->query("DELETE FROM printerAssoc WHERE paID=0 OR paHostID=0 OR paPrinterID=0");
// $this->DB->query("DELETE FROM printers WHERE pID=0");
// $this->DB->query("DELETE FROM pushbullet WHERE pID=0");
// $this->DB->query("DELETE FROM pxeMenu WHERE pxeID=0");
// $this->DB->query("DELETE FROM scheduledTasks WHERE stID=0 OR stTaskTypeID=0 OR stGroupHostID=0 OR stActive=0");
// $this->DB->query("DELETE FROM schemaVersion WHERE vID=0");
// $this->DB->query("DELETE FROM snapinAssoc WHERE saID=0 OR saHostID=0 OR saSnapinID=0");
// $this->DB->query("DELETE FROM snapinGroupAssoc WHERE sgaID=0 OR sgaSnapinID=0 OR sgaStorageGroupID=0");
// $this->DB->query("DELETE FROM snapinJobs WHERE sjID=0 OR sjHostID=0");
// $this->DB->query("DELETE FROM snapins WHERE snapinID=0");
// $this->DB->query("DELETE FROM snapinTasks WHERE stID=0 OR stJobID=0 OR stSnapinID=0");
// $this->DB->query("DELETE FROM supportedOS WHERE osID=0");
// $this->DB->query("DELETE FROM taskLog WHERE id=0 OR taskID=0");
// $this->DB->query("DELETE FROM tasks WHERE taskID=0 OR taskHostID=0 OR taskTypeID=0");
// $this->DB->query("DELETE FROM taskStates WHERE tsID=0");
// $this->DB->query("DELETE FROM taskTypes WHERE ttID=0");
// $this->DB->query("DELETE FROM torrent WHERE id=0");
// $this->DB->query("DELETE FROM userCleanup WHERE ucID=0");
}

I don’t think innodb can handle this big loop over and over.

This is the stuff that was showing on Joseph’s system over and over when he ran this command against mysql:show full processlist;

Latest version of SVN 4162 did remove host ping from my top 10 loads but did not fix the issue with the errors we were seeing and web pages not responding. MySQLCheck did not find any issues with the database itself.

FOGPingHosts works as a service similar to FOGMulticastManager or FOGImageReplicator. It’s purpose is to ping the hosts and report the “status” so that you’re not waiting on the List All/Search on the GUI to move between pages.

It pings each host and updates the ping status column on the database. This iterates every 5 minutes after the last one updates. So every 5 minutes it’s pinging all hosts of the server.

Now I’ve added a usleep of 1/1000th of a second per each host ping to allow the CPU to cool down between themselves so we don’t completely overload your CPU. I’ll up this value just to help slow things down a bit to 1/10th of a second. I’m not going to adjust the sleep time as 5 minutes is okay for most, though you can adjust your value in the Config.class.php.

This would explain the mysql usage AND the FOGPingHosts taking up most of the time.

@Joseph-Hales ping hosts could be a culprit in that it pings all of your hosts as a service separately from the main. The ping time happens every five minutes after the last completion. My guess is this IS the sole reason to the problem you’re seeing considering from what I recall you have about 6k hosts. You can disable the service and my guess is you’ll see much better stays overall on your server, including MySQL.