tl;dr: the Alioth service is back online, on a new host. Make sure you still
have your data.
What went wrong?
Most of the data on vasks was stored on a RAID5. One of the disks in the RAID
experienced a small hiccup, leading to it being thrown from the RAID, at which
point mdadm started recovery using a hotspare. During this, another disk in the
RAID threw hard errors and we were stuck with a non-working RAID.
Investigations showed this did not affect the /srv partition, but it affected
the PostgreSQL partition. Due to various misconfigurations, we did not have any
recent backups of Postgres.
How it got fixed
Level-28 black voodoo Perl magic by mjt, to extract the contents of a failed
RAID array that mdadm would barf on, allowed us to restore the database.
Alioth went from a dual-head setup, where part of the services were on
vasks.debian.org and part were on wagner.d.o, to a single, more powerful,
server called moszumanska.d.o. Much of the data was simply copied over, and
some manual reconfiguration was needed to adapt to that setup.
You don't want to type that hostname, so please use the functional aliases
(git.d.o, alioth.d.o, and so on).
What's left to do for you, dear user
· Update your SSH known keys. You can find all debian.org ssh hostkeys
at [1] or on any debian.org system in /etc/ssh/ssh_known_hosts. Or
you could just use the fingerprints in DNS (secured by DNSSEC).
· Check your homedir. It should have been copied over from vasks,
but make sure nothing is missing. Also, please take the opportunity to
clean up obsolete stuff, as we're often bothered by space constraints.
· Look in /mnt/home/users, which has a copy of what was on wagner,
and grab what you need from there. Note that this mountpoint will
disappear soon (in a few weeks at most), so don't delay. Please only
copy what's still relevant, for the disk space considerations mentioned
above.
· Check your scripts and websites. We remind you that Alioth is not
meant to be a generic hosting facility, so if you do advanced stuff
you're on your own.
· Use this opportunity to evaluate your needs and your usage. Alioth
is not meant to be a generic hosting facility. Probably much of what
is currently hosted on alioth is better suited to living somewhere else,
like people.d.o (or a debian.net server, or a debian.org server). If
this applies to your project, please get in touch with DSA [2].
· Copy over your crontabs from vasks/wagner.
· Check your repositories.
Cheers,
--
-----------------------------------------------------------------
| ,''`. Stephen Gran |
| : :' : sgran@debian.org |
| `. `' Debian user, admin, and developer |
| `- http://www.debian.org |
-----------------------------------------------------------------
[1] https://db.debian.org/debian_known_hosts
[2] http://dsa.debian.org/