My inital analysis shows that it was a kernel bug. At 14:36:31, July 7th, a kernel error was logged. Shortly after the inital, another was logged, and ~ 20 seconds later mysql restarted and logged an error. The kernel error was logged several more times, and mysqld restarted once more in an attempt to stay alive.

Klieber notified me, and at that time it had appeared as if only mysqld had died. I tried shutting services, to no avail. I then attempted to reboot the server, and it never came back.

Klieber swung by the data center earlier this morning and says that server was locked up at the console. I imagine that the kernel "oops" struck again in the shutdown sequence. Hard reboot restored services.

In light of the crash, their appears to be no data loss, and the server is back up running fine. The kernel will get replaced soon. _________________- Kyle Manna