Recommended Posts

The eFestivals forums are not currently available. Apologies for any inconvenience.

At around 1am today, the main eFestivals server suffered a disk failure. We are currently working on restoring the forums content onto a new server from our backups.

The forums should be back on-line at around 6pm this evening.

Unfortunately, the majority of posts from Sat 4th Apr will probably have been lost, as the last backup was taken at around 5am that day. However, it is still possible that these might be recoverable, as we're still looking at what might be recoverable from the damaged disk.

Thought it was busy this morning due to belated Glasto-frenzy. Hope it can be fixed easily.

Share this post

Link to post

Share on other sites

Good luck - am curious to hear what it was (if you can tell us) and what the consequences are.

on the main eFestivals server there were two physical disks, with one being a mirror of the other. One totally failed at just after 1am on Sunday morning, causing everything to then run off the second disk. However, it seems to be the case that there were already problems with that second disk, which then started to fail more catastrophically as it was doing more work.

At around 8:30am yesterday, I found the forums saying "too many connections" (I guess because some of the forums data was on the damaged part of that second disk, so inaccessible), while the main part of the website was still working.

In normal circumstances, a reboot of that server would have cleared the 'too many connections' problem, but I couldn't log onto the server to reboot it, so had to call the data centre support. They tried re-booting the server, but it wouldn't restart, because some of the operating system was on the damaged part of the disk. They tried various things to try and recover this disk, but without luck.

Meanwhile, they built a new server for me, which came online at around 1pm, and I then started to upload the backups onto that new server. However - and somewhat stupidly (lesson learnt ) - the backups were only here in my office, when they could have been on the other eFestivals servers as well .... because of the slow upload speed with ADSL, it took a long while for the first backups to be uploaded (if they'd have been on the other servers, it would have only taken minutes to copy them across). While they were uploading, I decided to not spoil my whole Sunday, and went to the pub to watch the footie.

Once the backups of the forums had uploaded, I was quickly able to restore these, and the forums were back online soon after 6pm yesterday. Because the last forums database backup (which has all of the posts and user data) had been made at around 5am on Saturday morning, that was the best backup that could be reverted to. Tha last backup of the forums directory - which includes user avatars, as well as the forums galleries - had been made in mid-march, so a few weeks of avatar changes &/or new user photos have been lost.

None of the content of the main part of the website has been lost. The pages, listings, etc data is held in databases on a different server, so this was completely unaffected. There was an issue with the setup of the new server tho, which meant I couldn't get the new server to access this data on that other server - this issue was resolved before 10am this morning, and the main part of the site has had full functionality (tho without photos) since then.

Meanwhile, since around 1pm yesterday, the backup of the photos has been being uploaded - this will take something like 55 hours to upload, which means that the vast majority of the photos won't be re-available until sometime late tomorrow. In the meantime, various adhoc photos have been uploaded, so that the front page looks complete.

Unfortunately, the last backup of the photos was done just before xmas, so the backup won't be able to restore those photos. However, we do have copies of those photos, so these will have to be put back manually, which isn't a huge task as there's only been a few festies covered so far this year.

While all this has been a big hassle and it's a shame that the server failed on an important day (Glasto ticket day), it's been far from a disaster, as just about everything is recoverable even tho it'll take a little while to get things straight again. Lessons have of course been learned, meaning that the way backups are made will be improved so that if similar happens again things will be a little easier to restore.

Share this post

Link to post

Share on other sites

Thanks, can follow most of that. Yup, front page hardly looks different, just a few photos that I've not seen before. Rumours pages seem to work fine again too. Had a few emails notifications to the forums that don't link correctly, but that's hardly a problem and it will right itself now things are working again.