tag:www.packagecloudstatus.io,2005:/historypackagecloud.io Status - Incident History2018-03-19T01:02:35-07:00packagecloud.iotag:www.packagecloudstatus.io,2005:Incident/15144072017-11-23T08:22:42-08:002017-11-23T08:22:42-08:00Degraded Performance<p><small>Nov 23, 08:22 PST</small><br><strong>Resolved</strong> - Increased capacity to help with load issues/degraded performance</p>tag:www.packagecloudstatus.io,2005:Incident/13882822017-09-28T20:53:38-07:002017-09-28T20:53:38-07:00Increased Load/Timeouts<p><small>Sep 28, 20:53 PDT</small><br><strong>Resolved</strong> - Capacity has been increased, resolved</p><p><small>Sep 28, 20:17 PDT</small><br><strong>Identified</strong> - Increased load is causing issues</p>tag:www.packagecloudstatus.io,2005:Incident/13455752017-09-01T11:12:30-07:002017-09-01T11:12:30-07:00Corrupted Deployment<p><small>Sep 1, 11:12 PDT</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Sep 1, 11:11 PDT</small><br><strong>Identified</strong> - The issue has been identified and a fix is being implemented.</p>tag:www.packagecloudstatus.io,2005:Incident/12935082017-07-13T23:29:53-07:002017-07-13T23:29:53-07:00Host Down/Unresponsive<p><small>Jul 13, 23:29 PDT</small><br><strong>Resolved</strong> - Everything is back to normal</p><p><small>Jul 13, 09:10 PDT</small><br><strong>Monitoring</strong> - Fully failed over, catching up on unprocessed jobs</p><p><small>Jul 13, 08:20 PDT</small><br><strong>Identified</strong> - Failing over</p>tag:www.packagecloudstatus.io,2005:Incident/10889362017-01-10T14:18:00-08:002017-01-10T14:18:26-08:00Degraded S3 Uploads<p><small>Jan 10, 14:18 PST</small><br><strong>Resolved</strong> - All affected repositories have been reindexed and uploads appear to be working normally.</p><p><small>Jan 10, 14:09 PST</small><br><strong>Monitoring</strong> - AWS has rolled out a fix and all affected repositories are being reindexed.</p><p><small>Jan 10, 13:20 PST</small><br><strong>Identified</strong> - AWS has confirmed that the issue is internal to AWS and they are working on deploying a fix.</p><p><small>Jan 10, 13:09 PST</small><br><strong>Update</strong> - In contact with AWS support to identify and resolve issue.</p><p><small>Jan 10, 12:56 PST</small><br><strong>Investigating</strong> - Amazon S3 is throwing intermittent 500 Internal Server Errors on upload.</p>tag:www.packagecloudstatus.io,2005:Incident/6531292016-07-21T11:46:00-07:002016-07-21T11:46:01-07:00Hardware Failure<p><small>Jul 21, 11:46 PDT</small><br><strong>Resolved</strong> - Everything is up and running normally</p><p><small>Jul 21, 11:40 PDT</small><br><strong>Monitoring</strong> - Failover complete.</p><p><small>Jul 21, 11:32 PDT</small><br><strong>Identified</strong> - Currently failing over.</p>tag:www.packagecloudstatus.io,2005:Incident/5533612016-07-06T12:32:16-07:002016-07-06T12:32:17-07:00Looking into increased connection timeouts<p><small>Jul 6, 12:32 PDT</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Jul 6, 12:21 PDT</small><br><strong>Investigating</strong> - We are currently investigating this issue.</p>tag:www.packagecloudstatus.io,2005:Incident/5139182016-05-23T14:02:55-07:002016-05-23T14:02:55-07:00Bad deploy has triggered some service failures<p><small>May 23, 14:02 PDT</small><br><strong>Resolved</strong> - Service has been restored.</p><p><small>May 23, 13:55 PDT</small><br><strong>Identified</strong> - A bad deploy has triggered some service failures, the code is being reverted and redeployed now.</p>tag:www.packagecloudstatus.io,2005:Incident/4889382016-05-02T11:45:38-07:002016-05-02T11:45:38-07:00Queues Backed Up<p><small>May 2, 11:45 PDT</small><br><strong>Resolved</strong> - Everything is now running smoothly, sorry for any delays!</p><p><small>May 2, 11:37 PDT</small><br><strong>Monitoring</strong> - Queue pressure has been relieved, queues are starting to move along</p><p><small>May 2, 10:34 PDT</small><br><strong>Update</strong> - Package uploads result in a reindex job being run to regenerate repository metadata. The index queues are currently backed up as we have sustained a large number of package uploads in a very, very short period of time. We are working on some fixes to relieve pressure on our indexer processes.</p><p><small>May 2, 10:28 PDT</small><br><strong>Investigating</strong> - Queues are slow/backing up</p>tag:www.packagecloudstatus.io,2005:Incident/4374042016-03-07T23:08:57-08:002016-03-07T23:08:58-08:00Performance issues and response times<p><small>Mar 7, 23:08 PST</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Mar 7, 22:52 PST</small><br><strong>Investigating</strong> - Performance issues and response times.</p>tag:www.packagecloudstatus.io,2005:Incident/4349172016-03-05T00:50:36-08:002016-03-05T00:50:36-08:00Increased Response Times<p><small>Mar 5, 00:50 PST</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Mar 4, 23:21 PST</small><br><strong>Investigating</strong> - Increased response times.</p>tag:www.packagecloudstatus.io,2005:Incident/3509722015-11-05T08:37:43-08:002015-11-05T08:37:43-08:00Potential hardware failure<p><small>Nov 5, 08:37 PST</small><br><strong>Resolved</strong> - Broken hardware was disabled and services have been restored. We are still monitoring the situation closely.</p><p><small>Nov 5, 07:58 PST</small><br><strong>Investigating</strong> - We are currently investigating a potential hardware failure and taking steps to fail over.</p>tag:www.packagecloudstatus.io,2005:Incident/1696912014-12-03T17:35:00-08:002015-02-07T17:44:17-08:00DNS Outage<p><small>Feb 7, 17:44 PST</small><br><strong>Postmortem</strong> - ## TL;DR
<br />
<br />This past Monday, our DNS provider, DNS Simple, experienced a distributed denial of service attack which took down their DNS resolution service.
<br />
<br />You can find more information about the DNS outage at our provider here.
<br />
<br />Our monitoring alerted us that there was a problem with domain resolution and we began investigating. Our DNS provider is both our registrar and our DNS provider, so there was, unfortunately, nothing that we could do during the outage.
<br />
<br />During this time, some customers were unable to resolve our domain name packagecloud.io. Customers who had our DNS cached or added an entry to their /etc/hosts file were unaffected by the outage.
<br />
<br />We’ve made some changes to help mitigate our DNS provider having an outage in the future.
<br />
<br />## More info
<br />We were alerted by our monitoring services at 19:21 UTC on December 1, 2014 that DNS resolution was failing.
<br />
<br />We immediately began investigating the issue and found that DNS Simple was experiencing a distributed denial of service attack.
<br />
<br />You can find more information about the DNS outage at our provider here.
<br />
<br />Our DNS provider is both our registrar and our DNS provider. Their service was down in its entirety, so we were unable to login to switch our namserver settings to an alternate provider during the outage.
<br />
<br />Customers with our DNS cached on their systems were unaffected by the outage and we saw several customers downloading and uploading packages during the DNS outage.
<br />
<br />Once the service at our DNS provider was restored, we made some changes to help mitigate potential outages like this in the future.
<br />
<br />## Changes
<br />It’s possible to configure your DNS settings to use more than one provider to protect against a particular DNS provider having an outage.
<br />
<br />In order to do this, you need two DNS providers which support DNS zone transfers.
<br />
<br />We researched our options and selected two providers that support DNS zone transfers, migrated our DNS zones to the new providers, and updated our nameservers at our DNS registrar.
<br />
<br />We sincerely apologize for the outage our customers experienced and hope that the changes we made to our infrastructure help protect customers against future outages of this nature.
<br />
<br />If you have any questions, please feel free to email us at support@packagecloud.io.</p><p><small>Dec 3, 17:35 PST</small><br><strong>Resolved</strong> - Denial of Service on DNSimple causing the packagecloud.io domain to intermittently resolve.</p>