Wednesday, January 8, 2014

Multiple Hard Drive Faliure

We've lost several of the drives from the machine that hosts PAUSE. It's offline while we work to replace the drives and convince it to come back up.

Update 1/8/2013 5:49pm EST: No good news so far. We may need to restore from a backup. <distraction>Did you hear that github was down?</distraction>

Update 1/10/2013 1:26am EST: We're working to get PAUSE back up as soon as we can and apologize for any inconvenience this outage may be causing you. It is taking a little longer than normal due to circumstances that are limiting our availability this week. (i.e. bad timing for a failure.)

Hi everyone – the failure had particular bad timing in that Robert and I were both traveling when it happened.

Robert did the initial debugging and attempts to recover the drives remotely with our friends at http://www.yellowbot.com/ doing the physical work.

When we gave up on that I was back in California and started copying the latest backup to a newly built virtual machine.

When all that was done Andreas re-indexed and otherwise fixed the database delta between the last backup and when the box crashed using data from the master CPAN mirror and I am guessing the modules mailing list archive.