tag:status.feeder.co,2005:/historyfeeder.co Status - Incident History2018-02-02T22:31:12+01:00feeder.cotag:status.feeder.co,2005:Incident/15883122018-02-02T22:31:12+01:002018-02-02T22:31:12+01:00Service outage<p><small>Feb 2, 22:31 CET</small><br><strong>Resolved</strong> - Emergency mainenence is over. Sorry for any inconvenience.</p><p><small>Feb 2, 21:26 CET</small><br><strong>Monitoring</strong> - Service has been restored for a while now. However emergency maintenance will start soon causing downtime for 30 minutes.</p><p><small>Feb 2, 18:05 CET</small><br><strong>Identified</strong> - A database server is having hardware issues and needs to be migrated to a fresh host. We're waiting for more information from our provider.</p><p><small>Feb 2, 17:47 CET</small><br><strong>Investigating</strong> - A database server is not responding, causing the service to be unavailable. We're investigating</p>tag:status.feeder.co,2005:Incident/15694202018-01-19T04:00:00+01:002018-01-18T11:21:43+01:00Brief disruption of feed crawling<p><small>Jan 18, 11:21 CET</small><br><strong>Scheduled</strong> - At 2018-01-19 03:00 AM our hosting provider will reboot a server that has a key role in the feed crawling service. This means that feed crawling will be delayed for 15-30 minutes while this happens.
<br />
<br />This will be the final disruption for now in this string of maintenance. Our hosting provider expects further action to be needed in the mitigation of Spectre and Meltdown, but for now the biggest and most urgent issues will be mitigated. If you have questions please contact us at support@feeder.co</p>tag:status.feeder.co,2005:Incident/15683532018-01-18T06:00:00+01:002018-01-17T13:51:56+01:00Partial Outage while posts database reboots<p><small>Jan 17, 13:51 CET</small><br><strong>Scheduled</strong> - As part of the ongoing mitigation of the Spectre and Meltdown attacks (https://meltdownattack.com), our hosting provider has scheduled a reboot of one of our database servers. During that time new feeds won't be crawled and posts won't load properly. The reboot has been scheduled for 2018-01-18 05:00 AM UTC and we estimate it will take around 30 minutes.
<br />
<br />Again, we're really sorry for the short notice. We're notifying you as fast as our hosting provider schedules these windows.</p>tag:status.feeder.co,2005:Incident/15668092018-01-17T06:36:26+01:002018-01-17T06:36:26+01:00Full Service Outage while main database is restarted<p><small>Jan 17, 06:36 CET</small><br><strong>Completed</strong> - Everything is back up and running now. Thanks for your patience! If you have any questions: Please contact support@feeder.co</p><p><small>Jan 16, 09:46 CET</small><br><strong>Scheduled</strong> - Our main database server has been scheduled for a restart tomorrow at 2018-01-17 05:00 AM UTC, meaning full and total downtime. We're expecting it to take around 15 minutes. We're sorry for the inconvenience and short notice, but must push forward to continue to ensure our infrastructure is secured against Meltdown and Spectre. (meltdownattack.com)</p>tag:status.feeder.co,2005:Incident/15655152018-01-16T09:44:16+01:002018-01-16T09:44:16+01:00Restart of our entire infrastructure during the day<p><small>Jan 16, 09:44 CET</small><br><strong>Completed</strong> - Mondays scheduled maintenance is over. There are still some updates that need to be applied by our hosting provider. We'll publish separate notices on our status page for that.</p><p><small>Jan 16, 07:23 CET</small><br><strong>Update</strong> - A restart of one of the database servers is currently causing intermittent downtime of the servers. Everything should be back to normal in around 5 minutes.</p><p><small>Jan 14, 23:00 CET</small><br><strong>Scheduled</strong> - Tomorrow, Monday 15th of January, we will be restarting all servers in our infrastructure to update to the latest patches regarding Spectre and Meltdown. This means at some points during the day, some parts of the service might run a bit slower, or be unavailable for short periods of time. We're sorry for the short notice, but our hosting provider just made public their plans for this. We'll keep you posted when we have more information.</p>tag:status.feeder.co,2005:Incident/15230742017-12-01T17:24:16+01:002017-12-01T17:24:16+01:00Brief disruption of notifications<p><small>Dec 1, 17:24 CET</small><br><strong>Resolved</strong> - Between 16:00-17:15 today the server that sends out notifications and e-mail notifications encountered a hardware failure. It has been resolved by the service provider at this time and everything is back up and running. To avoid this type of downtime in the future we are looking into adding another server for redundancy.</p>tag:status.feeder.co,2005:Incident/14968182017-11-11T09:23:31+01:002017-11-11T09:23:31+01:00Maintenance of post database<p><small>Nov 11, 09:23 CET</small><br><strong>Completed</strong> - Done!</p><p><small>Nov 11, 08:00 CET</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Nov 9, 20:43 CET</small><br><strong>Scheduled</strong> - We need to upgrade 1 of our 2 post database servers. During this time we'll stop the crawling of feeds until we're done. Posts will not show up properly in the apps and web, but will come back once we're done.
<br />
<br />Afterwards we'll crawl everything and fetch and notify you about the stuff you missed!</p>tag:status.feeder.co,2005:Incident/13262192017-08-12T07:18:39+02:002017-08-12T07:18:39+02:00Major outage<p><small>Aug 12, 07:18 CEST</small><br><strong>Resolved</strong> - The issue has been found and fixed and everything is back up now.</p><p><small>Aug 12, 06:52 CEST</small><br><strong>Investigating</strong> - Our database server is down due to a physical issue. We are investigating.</p>tag:status.feeder.co,2005:Incident/12572142017-06-02T15:50:00+02:002017-06-02T16:13:43+02:00Post are not loading<p><small>Jun 2, 15:50 CEST</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Jun 2, 15:41 CEST</small><br><strong>Monitoring</strong> - Database server is up. Monitoring the status right now</p><p><small>Jun 2, 15:37 CEST</small><br><strong>Investigating</strong> - One of database servers stalled and we are investigating.</p>tag:status.feeder.co,2005:Incident/12550262017-05-31T14:25:22+02:002017-05-31T14:25:22+02:00Feed Crawlers are stopped due to emergency maintenence<p><small>May 31, 14:25 CEST</small><br><strong>Resolved</strong> - The duplicate posts have been removed and everything is back to normal. We apologize for any inconvenience called, and if you are seeing duplicate posts still please contact support@feeder.co</p><p><small>May 31, 12:14 CEST</small><br><strong>Monitoring</strong> - The database for feed IDs is back up and crawlers are enabled. We're removing duplicate posts that came up during the incident.</p><p><small>May 31, 08:47 CEST</small><br><strong>Update</strong> - The database in question is just the database containing all feed IDs that are used to identify duplicates. This is why some people saw a large amount of duplicate posts before we stopped the crawlers.</p><p><small>May 31, 08:39 CEST</small><br><strong>Update</strong> - Database recovery is at 10% currently, been running for 5 minutes.</p><p><small>May 31, 08:29 CEST</small><br><strong>Identified</strong> - A database error caused duplicates for new posts. We have stopped the feed crawlers while we are fixing the database</p>tag:status.feeder.co,2005:Incident/11416702017-02-28T22:34:41+01:002017-02-28T22:34:41+01:00Posts are not showing and e-mails are not going out<p><small>Feb 28, 22:34 CET</small><br><strong>Resolved</strong> - Amazon seem to have fixed most of the issues now for reading posts. We are marking this as resolved. Please contact us if you are experiencing issues: support@feeder.co</p><p><small>Feb 28, 19:00 CET</small><br><strong>Identified</strong> - AWS appears to be having issues in our region regarding S3 and SES (Simple E-mail Service). There is nothing we can do but ride out this wave. Follow #aws on twitter for the action.</p>tag:status.feeder.co,2005:Incident/10235662016-12-15T04:51:31+01:002016-12-15T04:51:31+01:00Database server down<p><small>Dec 15, 04:51 CET</small><br><strong>Resolved</strong> - The hardware provider has resolved the issue.
<br />
<br />This was the second time in just 16 days this happened on our most valuable server (the database). This is unacceptable and we will be looking into alternative database hosting.</p><p><small>Dec 15, 04:31 CET</small><br><strong>Identified</strong> - Our database server is down due to a hardware issue on the physical host. Our provider is investigating.</p>tag:status.feeder.co,2005:Incident/10018642016-11-28T13:57:45+01:002016-11-28T13:57:45+01:00Feeder is down<p><small>Nov 28, 13:57 CET</small><br><strong>Resolved</strong> - Our database server is back up and we're combing through everything to make sure nothing is weird. Feel free to contact support@feeder.co if you have any questions.</p><p><small>Nov 28, 13:09 CET</small><br><strong>Identified</strong> - Our hosting provided has confirmed there's a problem on the physical hardware. We are awaiting their response.</p><p><small>Nov 28, 13:07 CET</small><br><strong>Investigating</strong> - Our database server has lost connection to the world. We're looking into it.</p>tag:status.feeder.co,2005:Incident/9817662016-11-07T13:04:58+01:002016-11-07T13:04:58+01:00Database Downtime<p><small>Nov 7, 13:04 CET</small><br><strong>Resolved</strong> - A heavy query caused the database to crash due to Out of Memory issues. To try to resolve this I restarted the database, which would then not go back up. After some investigation a bad configuration value was keeping the database from booting. It fixed and the service is back up. We're really sorry about this, please contact support@feeder.co if you have further questions.</p>tag:status.feeder.co,2005:Incident/9440032016-10-09T14:50:57+02:002016-10-09T14:50:58+02:00Connectivity issues<p><small>Oct 9, 14:50 CEST</small><br><strong>Resolved</strong> - A slow query was blocking the database. We aborted the query and everything is back up. We have added additional logging and will monitor this to understand why it happened.</p><p><small>Oct 9, 14:46 CEST</small><br><strong>Investigating</strong> - Feeder.co is not responding. We are looking into why.</p>tag:status.feeder.co,2005:Incident/9323022016-10-03T12:55:36+02:002016-10-03T12:55:36+02:00Database restart<p><small>Oct 3, 12:55 CEST</small><br><strong>Resolved</strong> - Everything should be back up at this time. Please e-mail support@feeder.co if you are seeing anything odd.</p><p><small>Oct 3, 12:53 CEST</small><br><strong>Identified</strong> - Due to an upgraded we had to do an emergency restart of the database server. Everything should be back up within a couple of minutes.</p>tag:status.feeder.co,2005:Incident/9249812016-09-29T14:00:00+02:002016-09-29T14:00:00+02:00feeder.co unavailable<p>The feeder.co domain and feeder crawler services were unavailable due to a database outage. A blog post is coming up on our blog about the incident.</p>