feeder.co Status - Incident Historyhttp://status.feeder.co
StatuspageFri, 02 Feb 2018 22:31:12 +0100Service outage
<p><small>Feb 2, 22:31 CET</small><br><strong>Resolved</strong> - Emergency mainenence is over. Sorry for any inconvenience.</p><p><small>Feb 2, 21:26 CET</small><br><strong>Monitoring</strong> - Service has been restored for a while now. However emergency maintenance will start soon causing downtime for 30 minutes.</p><p><small>Feb 2, 18:05 CET</small><br><strong>Identified</strong> - A database server is having hardware issues and needs to be migrated to a fresh host. We're waiting for more information from our provider.</p><p><small>Feb 2, 17:47 CET</small><br><strong>Investigating</strong> - A database server is not responding, causing the service to be unavailable. We're investigating</p> Fri, 02 Feb 2018 22:31:12 +0100http://status.feeder.co/incidents/1zbbw7gkt579
http://status.feeder.co/incidents/1zbbw7gkt579Brief disruption of feed crawling
<p><small>Jan 18, 11:21 CET</small><br><strong>Scheduled</strong> - At 2018-01-19 03:00 AM our hosting provider will reboot a server that has a key role in the feed crawling service. This means that feed crawling will be delayed for 15-30 minutes while this happens.
<br />
<br />This will be the final disruption for now in this string of maintenance. Our hosting provider expects further action to be needed in the mitigation of Spectre and Meltdown, but for now the biggest and most urgent issues will be mitigated. If you have questions please contact us at support@feeder.co</p> Fri, 19 Jan 2018 04:00:00 +0100http://status.feeder.co/incidents/gptvwtd3zwnd
http://status.feeder.co/incidents/gptvwtd3zwndPartial Outage while posts database reboots
<p><small>Jan 17, 13:51 CET</small><br><strong>Scheduled</strong> - As part of the ongoing mitigation of the Spectre and Meltdown attacks (https://meltdownattack.com), our hosting provider has scheduled a reboot of one of our database servers. During that time new feeds won't be crawled and posts won't load properly. The reboot has been scheduled for 2018-01-18 05:00 AM UTC and we estimate it will take around 30 minutes.
<br />
<br />Again, we're really sorry for the short notice. We're notifying you as fast as our hosting provider schedules these windows.</p> Thu, 18 Jan 2018 06:00:00 +0100http://status.feeder.co/incidents/mww237d1btgf
http://status.feeder.co/incidents/mww237d1btgfFull Service Outage while main database is restarted
<p><small>Jan 17, 06:36 CET</small><br><strong>Completed</strong> - Everything is back up and running now. Thanks for your patience! If you have any questions: Please contact support@feeder.co</p><p><small>Jan 16, 09:46 CET</small><br><strong>Scheduled</strong> - Our main database server has been scheduled for a restart tomorrow at 2018-01-17 05:00 AM UTC, meaning full and total downtime. We're expecting it to take around 15 minutes. We're sorry for the inconvenience and short notice, but must push forward to continue to ensure our infrastructure is secured against Meltdown and Spectre. (meltdownattack.com)</p> Wed, 17 Jan 2018 06:36:26 +0100http://status.feeder.co/incidents/wr53sntgxzcq
http://status.feeder.co/incidents/wr53sntgxzcqRestart of our entire infrastructure during the day
<p><small>Jan 16, 09:44 CET</small><br><strong>Completed</strong> - Mondays scheduled maintenance is over. There are still some updates that need to be applied by our hosting provider. We'll publish separate notices on our status page for that.</p><p><small>Jan 16, 07:23 CET</small><br><strong>Update</strong> - A restart of one of the database servers is currently causing intermittent downtime of the servers. Everything should be back to normal in around 5 minutes.</p><p><small>Jan 14, 23:00 CET</small><br><strong>Scheduled</strong> - Tomorrow, Monday 15th of January, we will be restarting all servers in our infrastructure to update to the latest patches regarding Spectre and Meltdown. This means at some points during the day, some parts of the service might run a bit slower, or be unavailable for short periods of time. We're sorry for the short notice, but our hosting provider just made public their plans for this. We'll keep you posted when we have more information.</p> Tue, 16 Jan 2018 09:44:16 +0100http://status.feeder.co/incidents/b2g4zx0kl56r
http://status.feeder.co/incidents/b2g4zx0kl56rBrief disruption of notifications
<p><small>Dec 1, 17:24 CET</small><br><strong>Resolved</strong> - Between 16:00-17:15 today the server that sends out notifications and e-mail notifications encountered a hardware failure. It has been resolved by the service provider at this time and everything is back up and running. To avoid this type of downtime in the future we are looking into adding another server for redundancy.</p> Fri, 01 Dec 2017 17:24:16 +0100http://status.feeder.co/incidents/pt8r2y01hq22
http://status.feeder.co/incidents/pt8r2y01hq22Maintenance of post database
<p><small>Nov 11, 09:23 CET</small><br><strong>Completed</strong> - Done!</p><p><small>Nov 11, 08:00 CET</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Nov 9, 20:43 CET</small><br><strong>Scheduled</strong> - We need to upgrade 1 of our 2 post database servers. During this time we'll stop the crawling of feeds until we're done. Posts will not show up properly in the apps and web, but will come back once we're done.
<br />
<br />Afterwards we'll crawl everything and fetch and notify you about the stuff you missed!</p> Sat, 11 Nov 2017 09:23:31 +0100http://status.feeder.co/incidents/xnp06t9h4y1t
http://status.feeder.co/incidents/xnp06t9h4y1tMajor outage
<p><small>Aug 12, 07:18 CEST</small><br><strong>Resolved</strong> - The issue has been found and fixed and everything is back up now.</p><p><small>Aug 12, 06:52 CEST</small><br><strong>Investigating</strong> - Our database server is down due to a physical issue. We are investigating.</p> Sat, 12 Aug 2017 07:18:39 +0200http://status.feeder.co/incidents/p5cpty3fm232
http://status.feeder.co/incidents/p5cpty3fm232Post are not loading
<p><small>Jun 2, 15:50 CEST</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Jun 2, 15:41 CEST</small><br><strong>Monitoring</strong> - Database server is up. Monitoring the status right now</p><p><small>Jun 2, 15:37 CEST</small><br><strong>Investigating</strong> - One of database servers stalled and we are investigating.</p> Fri, 02 Jun 2017 15:50:00 +0200http://status.feeder.co/incidents/n9jt5kn8wcm6
http://status.feeder.co/incidents/n9jt5kn8wcm6Feed Crawlers are stopped due to emergency maintenence
<p><small>May 31, 14:25 CEST</small><br><strong>Resolved</strong> - The duplicate posts have been removed and everything is back to normal. We apologize for any inconvenience called, and if you are seeing duplicate posts still please contact support@feeder.co</p><p><small>May 31, 12:14 CEST</small><br><strong>Monitoring</strong> - The database for feed IDs is back up and crawlers are enabled. We're removing duplicate posts that came up during the incident.</p><p><small>May 31, 08:47 CEST</small><br><strong>Update</strong> - The database in question is just the database containing all feed IDs that are used to identify duplicates. This is why some people saw a large amount of duplicate posts before we stopped the crawlers.</p><p><small>May 31, 08:39 CEST</small><br><strong>Update</strong> - Database recovery is at 10% currently, been running for 5 minutes.</p><p><small>May 31, 08:29 CEST</small><br><strong>Identified</strong> - A database error caused duplicates for new posts. We have stopped the feed crawlers while we are fixing the database</p> Wed, 31 May 2017 14:25:22 +0200http://status.feeder.co/incidents/kpyb0m8js3f8
http://status.feeder.co/incidents/kpyb0m8js3f8Posts are not showing and e-mails are not going out
<p><small>Feb 28, 22:34 CET</small><br><strong>Resolved</strong> - Amazon seem to have fixed most of the issues now for reading posts. We are marking this as resolved. Please contact us if you are experiencing issues: support@feeder.co</p><p><small>Feb 28, 19:00 CET</small><br><strong>Identified</strong> - AWS appears to be having issues in our region regarding S3 and SES (Simple E-mail Service). There is nothing we can do but ride out this wave. Follow #aws on twitter for the action.</p> Tue, 28 Feb 2017 22:34:41 +0100http://status.feeder.co/incidents/q447ypzq0923
http://status.feeder.co/incidents/q447ypzq0923Database server down
<p><small>Dec 15, 04:51 CET</small><br><strong>Resolved</strong> - The hardware provider has resolved the issue.
<br />
<br />This was the second time in just 16 days this happened on our most valuable server (the database). This is unacceptable and we will be looking into alternative database hosting.</p><p><small>Dec 15, 04:31 CET</small><br><strong>Identified</strong> - Our database server is down due to a hardware issue on the physical host. Our provider is investigating.</p> Thu, 15 Dec 2016 04:51:31 +0100http://status.feeder.co/incidents/lr67jt0wljwr
http://status.feeder.co/incidents/lr67jt0wljwrFeeder is down
<p><small>Nov 28, 13:57 CET</small><br><strong>Resolved</strong> - Our database server is back up and we're combing through everything to make sure nothing is weird. Feel free to contact support@feeder.co if you have any questions.</p><p><small>Nov 28, 13:09 CET</small><br><strong>Identified</strong> - Our hosting provided has confirmed there's a problem on the physical hardware. We are awaiting their response.</p><p><small>Nov 28, 13:07 CET</small><br><strong>Investigating</strong> - Our database server has lost connection to the world. We're looking into it.</p> Mon, 28 Nov 2016 13:57:45 +0100http://status.feeder.co/incidents/h6lz9cq8h6hy
http://status.feeder.co/incidents/h6lz9cq8h6hyDatabase Downtime
<p><small>Nov 7, 13:04 CET</small><br><strong>Resolved</strong> - A heavy query caused the database to crash due to Out of Memory issues. To try to resolve this I restarted the database, which would then not go back up. After some investigation a bad configuration value was keeping the database from booting. It fixed and the service is back up. We're really sorry about this, please contact support@feeder.co if you have further questions.</p> Mon, 07 Nov 2016 13:04:58 +0100http://status.feeder.co/incidents/w3w5t7hqfjkr
http://status.feeder.co/incidents/w3w5t7hqfjkrConnectivity issues
<p><small>Oct 9, 14:50 CEST</small><br><strong>Resolved</strong> - A slow query was blocking the database. We aborted the query and everything is back up. We have added additional logging and will monitor this to understand why it happened.</p><p><small>Oct 9, 14:46 CEST</small><br><strong>Investigating</strong> - Feeder.co is not responding. We are looking into why.</p> Sun, 09 Oct 2016 14:50:57 +0200http://status.feeder.co/incidents/wbv5846j22r4
http://status.feeder.co/incidents/wbv5846j22r4Database restart
<p><small>Oct 3, 12:55 CEST</small><br><strong>Resolved</strong> - Everything should be back up at this time. Please e-mail support@feeder.co if you are seeing anything odd.</p><p><small>Oct 3, 12:53 CEST</small><br><strong>Identified</strong> - Due to an upgraded we had to do an emergency restart of the database server. Everything should be back up within a couple of minutes.</p> Mon, 03 Oct 2016 12:55:36 +0200http://status.feeder.co/incidents/2pyfnqqdpsgx
http://status.feeder.co/incidents/2pyfnqqdpsgxfeeder.co unavailable
<p>The feeder.co domain and feeder crawler services were unavailable due to a database outage. A blog post is coming up on our blog about the incident.</p> Thu, 29 Sep 2016 14:00:00 +0200http://status.feeder.co/incidents/wshkvbh9htbq
http://status.feeder.co/incidents/wshkvbh9htbq