Post navigation

[Complete] Recovered from disk failures

A couple of our servers seems to be suffering from a CentOS kernel bug and disk faillures. We are going to take the service offline for a few hours to understand what is causing the problem and apply the necessary patches. Our goal is to get everything back to normal by the end of the week end. Sorry for the inconvenience. Note: if you need to urgently access your feeds, you can use the Google Reader interface: http://reader.google.com

Update: 09/25/2010 7:00pm PST – Disks swapped. Rebuild RAID configuration. Hiccups are expected during the next 24-36 hours until the RAID repair is complete. Thank you for your patience.

Update: 09/26/2010 11:00am PST – Things got worst during the night. A second disk failed during the night. We are in the process of rebuild a new machine based on the back up we performed yesterday. There will be a little bit of turbulence early this afternoon but we should be back up on brand new hardware by the end of the day.

Update: 09/26/2010 11:56am PST – Back online! – The dev team finished getting a new set of servers up and running and running (welcome to spyders). The service should be back only (and start feeling faster). We also pushed out a new version of the client: 3.4.305 which fixes some of the most important bugs reported over the last 4 weeks and offers a clean new integration with #newtwitter. You can get 3.x.305 from http://www.feedly.com Thanks for your patience and support during the last 24 hours! Kudos to Zol, Slushi and Merlin for killing this snake!

I will pass you thank you to the dev team: they worked around the clock over the last 24 hours to bring the new hardware online as fast as possible. Kudos for zol, slushi and merlin for the awesome firefighting!