tag:status.packet.net,2005:/historyPacket Status - Incident History2018-02-21T23:42:30-05:00Packettag:status.packet.net,2005:Incident/16065352018-02-16T15:52:11-05:002018-02-16T15:52:12-05:00EWR1 Unreachable Elastic IPs<p><small>Feb 16, 15:52 EST</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Feb 16, 14:05 EST</small><br><strong>Identified</strong> - The issues have been identified and we are working on a fix.</p><p><small>Feb 16, 13:45 EST</small><br><strong>Investigating</strong> - We are investigating reports of elastic IPs being unreachable to a subset of Type 0s in EWR1.</p>tag:status.packet.net,2005:Incident/15967262018-02-16T06:47:14-05:002018-02-16T06:47:15-05:00EWR1 Type 0 Network Upgrade<p><small>Feb 16, 06:47 EST</small><br><strong>Completed</strong> - Network upgrade completed.</p><p><small>Feb 16, 06:00 EST</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Feb 8, 15:52 EST</small><br><strong>Scheduled</strong> - On Friday, February 16th at 6:00 AM EST, we will be upgrading the software on several Type 0 top of rack switches in our EWR1 datacenter. This work is necessary to address several stability-impacting bugs we’ve encountered, as well as to provide additional capacity to support further growth of the Packet platform. Server provisions will be suspended in EWR1 during the maintenance window.
<br />
<br />This upgrade only affects a subset of our Type 0 server capacity and no other server types. Your Type 0 machine will be inaccessible on the network for up to thirty (30) minutes while we reload customer-attach switches. Machines will NOT lose power or reboot. If you have a block device attached via iSCSI to your Type 0 machine, the network timeout may be sufficiently long that a reboot may be the quickest way for you to re-attach your volume.
<br />
<br />If you'd like to get a specific list of the Type 0 machines of yours that will be impacted, please feel free to reach out to us at help@packet.net or live chat from our customer portal (https://app.packet.net).</p>tag:status.packet.net,2005:Incident/15975762018-02-09T11:30:44-05:002018-02-09T11:30:44-05:00Type0 Provisioning Issue<p><small>Feb 9, 11:30 EST</small><br><strong>Resolved</strong> - This issue is now resolved. Please reach out via help@packet.net, should you experience any issues.</p><p><small>Feb 9, 10:10 EST</small><br><strong>Identified</strong> - We have identified the issue and are working to put a fix in place.</p><p><small>Feb 9, 09:40 EST</small><br><strong>Investigating</strong> - Our team has identified an issue related to provisioning some Type 0 instances in all facilities. We will update the status as soon as a new information is available.</p>tag:status.packet.net,2005:Incident/15891802018-02-03T20:26:52-05:002018-02-03T20:26:53-05:00Website Outage<p><small>Feb 3, 20:26 EST</small><br><strong>Resolved</strong> - Issues with our website have been resolved. Apologies for the interruption!</p><p><small>Feb 3, 19:55 EST</small><br><strong>Investigating</strong> - We are investigating an issue with our website. The portal is still available via https://app.packet.net/</p>tag:status.packet.net,2005:Incident/15506812018-01-16T05:00:44-05:002018-01-16T05:00:44-05:00Portal and API Upgrade<p><small>Jan 16, 05:00 EST</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Jan 16, 04:01 EST</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Jan 2, 11:56 EST</small><br><strong>Scheduled</strong> - We will perform a one hour system-wide upgrade to deploy our redesigned portal and launch support for organizations in our API. During this time, our portal and API will be inaccessible.</p>tag:status.packet.net,2005:Incident/15604732018-01-10T02:03:57-05:002018-01-10T02:03:57-05:00Packet Portal is Slow<p><small>Jan 10, 02:03 EST</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Jan 9, 20:22 EST</small><br><strong>Monitoring</strong> - We have implemented a fix to the issue and are monitoring the results.</p><p><small>Jan 9, 19:56 EST</small><br><strong>Investigating</strong> - We are seeing issues with our client portal being slow. We're investigating.</p>tag:status.packet.net,2005:Incident/15603892018-01-09T19:04:12-05:002018-01-09T19:04:13-05:00Packet Portal is Slow<p><small>Jan 9, 19:04 EST</small><br><strong>Resolved</strong> - We have resolved the issue affecting response times from our portal and API.</p><p><small>Jan 9, 18:09 EST</small><br><strong>Monitoring</strong> - We have resolved the issue and are continuing to monitor.</p><p><small>Jan 9, 18:03 EST</small><br><strong>Investigating</strong> - We are seeing issues with our client portal and API being slow. We're investigating.</p>tag:status.packet.net,2005:Incident/15515952018-01-03T22:49:07-05:002018-01-03T22:49:07-05:00SJC1 - Rack Network Outage<p><small>Jan 3, 22:49 EST</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Jan 3, 12:05 EST</small><br><strong>Monitoring</strong> - We are continuing to monitor the situation, however all outages should be resolved at this time.</p><p><small>Jan 3, 10:45 EST</small><br><strong>Investigating</strong> - We are troubleshooting connectivity issues impacting customers on a single rack in our Sunnyvale, CA (SJC1) datacenter.
<br />
<br />Additional updates will follow.</p>tag:status.packet.net,2005:Incident/15460612017-12-25T08:34:36-05:002017-12-25T08:34:37-05:00SJC1 provisioning / deprovisioning issue<p><small>Dec 25, 08:34 EST</small><br><strong>Resolved</strong> - Please be advised that provisioning is fully restored in SJC1 at this time. Production networking was not impacted during the incident.</p><p><small>Dec 25, 01:56 EST</small><br><strong>Identified</strong> - Our team has identified an issue isolated to a single rack in SJC1 site. We will update the status as soon as a new information is available!</p>tag:status.packet.net,2005:Incident/15432312017-12-20T17:34:48-05:002017-12-20T17:34:48-05:00Provisioning Issues<p><small>Dec 20, 17:34 EST</small><br><strong>Resolved</strong> - The provisioning issue has been resolved and we are monitoring our provisioning queues for any abnormalities.</p><p><small>Dec 20, 17:34 EST</small><br><strong>Monitoring</strong> - We have identified the issue and are confirming the resolution now.</p><p><small>Dec 20, 17:27 EST</small><br><strong>Identified</strong> - We have identified the issue and are confirming the resolution now.</p><p><small>Dec 20, 17:08 EST</small><br><strong>Investigating</strong> - We are investigating provisioning issues in several of our global datacenters.</p>tag:status.packet.net,2005:Incident/15095542017-11-19T13:40:51-05:002017-11-19T13:40:52-05:00Portal and API issues<p><small>Nov 19, 13:40 EST</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Nov 19, 11:45 EST</small><br><strong>Monitoring</strong> - A fix has been implemented and we are monitoring the results.</p><p><small>Nov 19, 10:55 EST</small><br><strong>Investigating</strong> - We are currently investigating an issue on the API, affecting the Portal and some new provisions.</p>tag:status.packet.net,2005:Incident/15061662017-11-16T05:17:59-05:002017-11-16T05:18:00-05:00FRA1 Provisioning<p><small>Nov 16, 05:17 EST</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Nov 16, 04:38 EST</small><br><strong>Investigating</strong> - Our team is investigating a provisioning issue within our Frankfurt, DE (FRA1) site. We will update this status with additional details momentarily.</p>tag:status.packet.net,2005:Incident/15035192017-11-14T15:23:34-05:002017-11-14T15:23:35-05:00Slow API response time.<p><small>Nov 14, 15:23 EST</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Nov 14, 13:19 EST</small><br><strong>Monitoring</strong> - A fix has been implemented and we are monitoring the results.</p><p><small>Nov 14, 12:16 EST</small><br><strong>Investigating</strong> - We are experiencing slower than normal response times from the API and portal. Device provisioning may be inconsistent as a result during this time, and users may receive errors when attempting to visit the portal or query our API.</p>tag:status.packet.net,2005:Incident/15029832017-11-14T07:41:57-05:002017-11-14T07:41:57-05:00SYD1 network Issue<p><small>Nov 14, 07:41 EST</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Nov 14, 06:25 EST</small><br><strong>Investigating</strong> - We are currently investigating a network related issue at SYD1 (Sydney), one of our Edge sites. Server provisioning at this location is currently affected.</p>tag:status.packet.net,2005:Incident/15030222017-11-14T07:08:27-05:002017-11-14T07:08:27-05:00Portal connection issue<p><small>Nov 14, 07:08 EST</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Nov 14, 06:57 EST</small><br><strong>Monitoring</strong> - The issue affecting our public portal has been fixed and service has been restored. We are continuing to monitor and will post any updates as necessary.</p><p><small>Nov 14, 06:44 EST</small><br><strong>Investigating</strong> - We are currently investigating an issue with connectivity to our public portal. Our public API and other services are unaffected.</p>tag:status.packet.net,2005:Incident/14872502017-11-07T02:02:25-05:002017-11-09T11:30:02-05:00NRT1 connectivity issue<p><small>Nov 9, 11:30 EST</small><br><strong>Postmortem</strong> - Incident: Top of Rack Switch Outage in Tokyo, JP (NRT1) Datacenter
<br />
<br />Outage Start Time: 12:25 AM (EST) on November 7, 2017
<br />Outage End Time: 1:30 AM (EST) on November 7, 2017
<br />
<br />Reason for Outage
<br />
<br />=Overview=
<br />
<br />During this period of time, some customers connected to a single cabinet/top of rack switch in our Tokyo (NRT1) datacenter experienced a loss of connectivity.
<br />
<br />=Identifying the Core Issue / Resolution=
<br />
<br />On investigation, we identified an issue with the software running on the device in question, which was resolved following a full reload.
<br />
<br />We are working with our hardware manufacturer, Juniper, at the highest levels, and have identified several flaws in their current code base related to how firewall Access Control Lists (ACLs) get programmed into the hardware, resulting in valid customer traffic getting discarded over time. With their assistance, we’ve also identified a code revision which may solve these issues, which we are currently testing in our lab. In addition, we’ve made some changes to our internal provisioning systems to help with how ACLs are generated and deployed, to help mitigate this issue.
<br />
<br />=Moving Forward=
<br />
<br />We are improving our internal procedures around incident response and network device troubleshooting, based upon the lessons learned from this particular outage. Though hardware failures are an unfortunate (and rare) fact of life, we endeavor to diagnose and recover from these issues as quickly as possible.
<br />
<br />As a customer, if you are interested in greater switch diversity, you can review the “switch ID” note, which is exposed in our API and customer portal for each device below the facility code on the server detail page.
<br />
<br />This (8) character field is an identifier for the physical (top of rack) switch each server is connected to, and dependent on for connectivity. Most instance types are available on multiple switches in each datacenter, and we are happy to work with you to promote greater switch diversity in your deployment if that is of interest. As we work to introduce some provisioning-time options around diversity, please don’t hesitate to drop us a note (help@packet.net) if we can assist.</p><p><small>Nov 7, 02:02 EST</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Nov 7, 01:43 EST</small><br><strong>Identified</strong> - Network has been restored, but our team is still working on the NRT1 provisioning. For the meantime, please hold on NRT1 deploy/install.</p><p><small>Nov 7, 00:53 EST</small><br><strong>Investigating</strong> - We are seeing an outage in NRT1 facility. Provisioning / Deprovisioning of devices and portal access via NRT1 is affected. Our team is already looking into this issue.</p>tag:status.packet.net,2005:Incident/14551692017-11-01T22:11:49-04:002017-11-01T22:11:49-04:00SJC1 Provisioning<p><small>Nov 1, 22:11 EDT</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Nov 1, 21:06 EDT</small><br><strong>Identified</strong> - There is an issue with the IP address assignment for newly created Projects. Customers should still be able to provision on their existing (active) Projects.
<br />
<br />Our team is currently working to resolve the issue.</p><p><small>Nov 1, 18:39 EDT</small><br><strong>Investigating</strong> - Our team is investigating a provisioning issue within our SJC1 facility. We will update this status with additional details momentarily.</p>tag:status.packet.net,2005:Incident/14227972017-10-18T10:53:45-04:002017-10-18T10:53:46-04:00SJC1 Network Issues<p><small>Oct 18, 10:53 EDT</small><br><strong>Resolved</strong> - The issue with the fiber provider has been resolved and stable since last night. We are closing this issue and will update with a brief RFO soon.</p><p><small>Oct 17, 15:15 EDT</small><br><strong>Monitoring</strong> - Today at around 2PM EST, we saw network issues in our SJC facility caused by an outage in one of our providers. The experienced issues might have been seen in the form of brief outages, which should have been resolved quickly.
<br />We are currently monitoring the situation.
<br />Please reach out at help@packet.net if you are still experiencing any related issues.</p>tag:status.packet.net,2005:Incident/13926552017-10-11T14:00:35-04:002017-10-11T14:00:36-04:00EWR1 Block Storage Upgrades<p><small>Oct 11, 14:00 EDT</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Oct 11, 12:00 EDT</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Oct 2, 15:20 EDT</small><br><strong>Scheduled</strong> - We will be upgrading our EWR1 block storage clusters to the latest point release. This is a minor upgrade to improve the stability of the underlying system and should not be customer impacting in any way. We anticipate no downtime.</p>tag:status.packet.net,2005:Incident/13926592017-10-10T14:00:55-04:002017-10-10T14:00:55-04:00AMS1 Block Storage Upgrades<p><small>Oct 10, 14:00 EDT</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Oct 10, 12:00 EDT</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Oct 2, 15:25 EDT</small><br><strong>Scheduled</strong> - We will be upgrading our AMS1 block storage clusters to the latest point release. This is a minor upgrade to improve the stability of the underlying system and should not be customer impacting in any way. We anticipate no downtime.</p>tag:status.packet.net,2005:Incident/13680382017-09-20T03:49:13-04:002017-09-20T03:49:14-04:00EWR1 Provisioning Failure<p><small>Sep 20, 03:49 EDT</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Sep 20, 03:42 EDT</small><br><strong>Update</strong> - This incident has been resolved.</p><p><small>Sep 20, 03:34 EDT</small><br><strong>Investigating</strong> - We are investigating a provisioning issue in EWR1.</p>tag:status.packet.net,2005:Incident/13598492017-09-14T14:09:49-04:002017-09-14T14:09:50-04:00Portal Issues<p><small>Sep 14, 14:09 EDT</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Sep 14, 13:58 EDT</small><br><strong>Monitoring</strong> - We have mitigated the issue and are now monitoring it before resolving.</p><p><small>Sep 14, 13:19 EDT</small><br><strong>Investigating</strong> - We are investigating a slowdown in our customer portal, with timeouts reported as well.</p>tag:status.packet.net,2005:Incident/13534882017-09-11T06:05:57-04:002017-09-11T06:05:58-04:00NRT1 Provisioning Failure<p><small>Sep 11, 06:05 EDT</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Sep 11, 05:55 EDT</small><br><strong>Investigating</strong> - We have identified an issue with the availability of IP(v4) public addresses in NRT1. Our Network Engineering is currently working on this issue.</p>tag:status.packet.net,2005:Incident/13476872017-09-06T18:09:18-04:002017-09-06T18:09:19-04:00AMS1 , SJC1 & EWR1 Block Storage Upgrades<p><small>Sep 6, 18:09 EDT</small><br><strong>Completed</strong> - Block storage has been upgraded successfully. Let us know if you see any issue help@packet.net</p><p><small>Sep 6, 13:00 EDT</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Sep 5, 11:35 EDT</small><br><strong>Scheduled</strong> - We will be upgrading our AMS1 , SJC1 & EWR1 block storage clusters to the latest point release. This is a minor upgrade to improve stability of the underlying system and should not be customer impacting in any way. We anticipate no downtime.</p>tag:status.packet.net,2005:Incident/13474662017-09-05T13:51:20-04:002017-09-05T13:51:20-04:00Provisioning Issues with Type 1's in EWR1<p><small>Sep 5, 13:51 EDT</small><br><strong>Resolved</strong> - Our engineering team has resolved this issue. Please reach out via help@packet.net should you continue to encounter issues.</p><p><small>Sep 5, 09:13 EDT</small><br><strong>Identified</strong> - We've identified the issue affecting some CentOS 7 provisions in EWR1 and will deploy a fix shortly.</p><p><small>Sep 5, 08:54 EDT</small><br><strong>Investigating</strong> - We are investigating an increase in failed provisions for Type 1 servers in our EWR1 facility</p>