Starting at about 1:30 PM (Pacific time) today, some of our internal database systems had an unexpected failure. This lead to problems with our control panel (“My Account”), support site, and blog. Some customers may have also had problems with some aspects of Webmail (in particular, with the address book).

No customer data or Web sites were affected, and no e-mail was lost.

All systems are running again, so no one should see any problems — please let us know if you do! Some things are running on backup systems, so we’re working on finishing up the fixes and restoring everything to its normal status.

This was an unplanned and fairly horrible (and embarrassing) problem. This is the first time our own account management database has completely failed in the 10 years we’ve been providing Web hosting service. Obviously, we consider this sort of thing to be unacceptable, and we sincerely apologize for any inconvenience caused.

As you may know, we perform nightly backups of each Web hosting customer’s data, including your Web site, databases, log files, and any e-mail stored on our servers. Until recently, however, you usually needed to write to our support team if you wanted to get something restored.

We’re pleased to announce that we have recently made it easier for you to access your backups. Now you can easily browse through all the backups available for your site and retrieve any files using a simple FTP or SSH connection. In addition, our control panel lets you quickly see exactly which backup sets are available.

As you probably know, we back up all of our servers every night. Our goal is to keep at least seven days of backups available, with additional older backups available where possible.

We’ve added a new page to our control panel showing the backups available for each account. To see it, just login to the control panel, then click Backups.

We make backups so that we can recover from unexpected occurrences such as data erasure or server failures. Of course, we also make the backups available to customers, because they can be a real life-saver when you need one. However, we want to remind customers that they should also make their own backups, especially if you need a different backup frequency or retention policy. Our Web page describing our backup system and policies has more details.