We’re getting reports of assets not loading for some users and we’ve been able to replicate this issue ourselves. It does not appear to be affecting all users so it might just be an issue with a particular node on the content delivery network. We are escalating the issue with them and will report back when we know more.

UPDATE – August 4th 2011 – 12:00AM EST – Our content delivery network provider has tracked this down to being a single node in their Washington, DC area cluster that’s causing the problem. We expect this issue to be resolved shortly and will update this post once we recieve confirmation from our provider that the issue is fixed.

UPDATE – August 4th 2011 – We received notice at approx 12:30AM EST this morning that this issue has been resolved.

UPDATE – August 4th 2011 – 9:25AM EST – We’re seeing different errors with reports of users getting 401 unauthorized errors when trying to load assets. We’re escalating this with our provider immediately and we apologize for the trouble.

UPDATE – August 4th 2011 – 4:45PM EST – Assets on blog pages and external sites should be working again – we have failed them over to our backup system while the content delivery network gets their act together. Unfortunately assets in RSS feeds are still not loading – we may fail those over to the backup systems soon as well. We sincerely apologize for these troubles today and will plan in the future to fail-over assets to backup systems much faster.

Note: having assets failed over to the backup system means newly uploaded assets may take a while before they begin to load properly.

UPDATE – August 4th 2011 – 7:30PM EST – All assets should now be serving correctly – we apologize for the problem and now have systems and procedures in place to prevent this sort of issue from occurring again.

In a continued effort to improve the reliability and speed of our services we are going to be moving all Libsyn user assets on to the same fast content delivery network that all our media files are currently served from. The content delivery network we use has 20 locations across the globe to deliver your content from. Serving your assets from this will allow them to be downloaded at blazing fast speeds just like your media files. This will speed up page loads on any site where you use the images, whether your own external site/feed or the Libsyn webpage and feed.

To complete the last steps of this move we need to put the Libsyn system into maintenance mode and disable FTP access. We have scheduled this maintenance for tomorrow, Tuesday July 19th. It will begin at 8AM when the system will go into maintenance mode and FTP will be disabled. As usual during these times your Libsyn webpage, feed and media files will continue to serve without interruption. We are giving ourselves a 1 hour window to complete the work so the expected completion time is 9AM.

We will update this post when the work begins and again when it is completed.

7/19/11 – 8AM EST – We have started work on the maintenance.

UPDATE: 8:40AM EST – You might see images not loading for a few minutes while the final changes propagate through our system.

UPDATE: 8:55AM EST – All maintenance work is completed we are turning the publishing app and FTP back on. Thanks for your patience.

We’ll be doing some maintenance on the FTP systems at approximately 1PM EST today. This will cause a very brief downtime in the FTP server during which users won’t be able to connect. We expect this downtime in the FTP server to be less than 5 minutes.

1:30PM EST – This maintenance was completed successfully with less than 5 minutes of FTP downtime as expected. If you notice any issues with FTP please e-mail support@libsyn.com

This morning while performing regularly scheduled updates on our servers there was a brief content delivery outage. This was an unintended outage and our developers worked immediately to resolve the issue. All issues have been resolved and we apologize for any inconvenience caused.

On Monday May 16th starting at 8AM we will maintenance on the stats system. This maintenance includes some pretty serious hardware upgrades to the two key statistics systems we have – reporting and processing. The first part of this maintenance will affect stats reporting – so you’ll likely see red error boxes when trying to view your stats. We expect this part to last at most 15 minutes. The majority of the work is to the stats processing system – this will cause stats processing to be paused for up 8 hours.

We will update this post throughout the maintenance period – noting when stats reporting is back on and then again when stats processing has been restarted and finally when stats processing is all caught up.

UPDATE – MAY 16 – 8AM EST – Maintenance window is beginning – stats reporting will be the first thing affected, you will see red error messages in the stats dashboard for roughly 15-20 minutes.

UPDATE – MAY 16 – 8:15AM EST – Stats reporting is back on and starts processing is now paused – it will be paused for up to 8 hours.

UPDATE – MAY 16 – 4:45PM EST – Stats processing is being turned back on – we will be watching it carefully and will make another post after processing gets caught up.

UPDATE – MAY 16 – 11:30PM EST – Stats processing is progressing well and we except them to be fully caught up sometime overnight.

We’ll be doing some system maintenance at 8AM EST on Monday. We expect this maintenance to be completed quickly but we are setting a 30 minute time window for it. During this time logins and FTP will be disabled, while media downloads, feeds and blog pages should be unaffected. We will update this post on Monday morning when we start the maintenance and then afterwards when it’s completed.

UPDATE [850 Eastern]: Update went smoothly, everything is up and running as expected. If you notice anything unusual, please contact support.

We are currently experiencing a delay receiving raw logs from our CDN, which has stopped stats processing. Check back for an update as soon as we find out when log delivery will be restored.

UPDATE: [1730 Eastern] CDN technicians have been looking into the problem since early this morning. We are still waiting to hear back from them about log delivery being restored.

UPDATE: [1820 Eastern] Our CDN has informed us that the issue was related to a hardware failure. They have restored logs from yesterday and today from an unaffected backup system and we are in the process of downloading these logs. Processing should commence within the next hour. Over the next day, stats will be backfilling and will be caught up as soon as possible. We appreciate your patience and apologize for the inconvenience. If there are any changes, we will update this post.

Today at 2PM EST we’ll be putting the system into maintenance mode for roughly 10 – 15 minutes. During that time we’ll be beginning some changes to our database infrastructure and automated failover processes that will prevent the issue that happened last night from occuring again.