I was moving around backup data today and had inadvertently connected the backup drives directly to mains power instead of the UPS system. In a peculiar mix of random events a nearby lightning strike took out power in a few seconds causing a kernel panic on the main server as the backup file systems abruptly disappeared from view during heavy I/O.
Normally this kind of unlucky events would incite the use of bad language in the local server room but in this case it turns out that the reboot fixed a problem that had been escalating since June 30th: A lot of Java threads had been spinning out of control, keeping the server under a constant 25% load. After the crash/reboot things are back to normal.

It turns out that the first runaway processes where probably caused by a leap second that was added in the night between June 30 and July 1st 2012. It coincides nicely with the first CPU drain starting.

Heh, always learning new stuff:
- Hook up backups through the UPS
- The time can actually be 23:59:60 UTC occasionally

Ah, BURP is back again finally. :-)
What was the problem for this long outage, Janus?
____________Life is Science, and Science rules. To the universe and beyond
Proud member of BOINC@HeidelbergMy BOINC-Stats