If this is your first visit, be sure to
check out the FAQ by clicking the
link above. You may have to register
before you can post: click the register link above to proceed. To start viewing messages,
select the forum that you want to visit from the selection below.

Actually, while it would have been a huge PITA, there are enough out-of-repository copies of the source floating around so that I'm sure they could have rebuilt the repos to at least the last-released state of the code base. Still not good, but not lost forever either.

Comment

This is the silent corruption that ZFS was created to prevent. Unfortunately, KDE's servers were not using it.

The real problem KDE sysadms overlooked is that mirroring is not backing up. ZFS wouldn't have saved them if their HDDs totally failed - ZFS is good at spotting hardware errors - that's true. But ZFS is not a magic bullet.

Comment

The real problem KDE sysadms overlooked is that mirroring is not backing up. ZFS wouldn't have saved them if their HDDs totally failed - ZFS is good at spotting hardware errors - that's true. But ZFS is not a magic bullet.

It is possible that the backups themselves would have been corrupted had the administrators been doing proper backups and these issues predated the fsck. In this situation, silent corruption caused a problem occurred because there was no way to recover after its effects had been felt.

ZFS is not a replacement for proper backups, but it makes doing them easier. End-to-end checksumming and self-healing would have prevented this mess before it happened had ZFS been deployed.

Comment

Unfortunately, proper backups would not have helped as much as one would think without a way to know whether or not the data was bad prior to doing them. It is not clear when the repositories became corrupted, although the blog post suggests that fsck was the trigger.