tag:status.c9.io,2005:/historyCloud9 IDE, LLC Status - Incident History2018-02-18T04:31:37ZCloud9 IDE, LLCtag:status.c9.io,2005:Incident/15319242017-12-11T05:00:20Z2017-12-11T05:00:20ZWorkspace connection issues<p><small>Dec 11, 05:00 UTC</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Dec 11, 04:44 UTC</small><br><strong>Investigating</strong> - Some users will be seeing workspace connection issues in the USW region. We're currently looking into what's causing this.</p>tag:status.c9.io,2005:Incident/15035992017-11-14T19:05:16Z2017-11-14T19:05:16ZIssues Loading Hosted Workspaces<p><small>Nov 14, 19:05 UTC</small><br><strong>Resolved</strong> - Fix has been pushed and has propagated to all hosts. If any Cloud9-hosted workspaces are still running into the following error:
<br />
<br />/mnt/shared/sbin/node
<br />ssh process died with exit code 255
<br />
<br />please reach out to support@c9.io so we can take a look. SSH workspaces should be unaffected.</p><p><small>Nov 14, 18:33 UTC</small><br><strong>Monitoring</strong> - Fix has been pushed and has propagated to all hosts. If any Cloud9-hosted workspaces are still running into the following error:
<br />
<br />/mnt/shared/sbin/node
<br />ssh process died with exit code 255
<br />
<br />please reach out to support@c9.io so we can take a look. SSH workspaces should be unaffected.</p><p><small>Nov 14, 18:12 UTC</small><br><strong>Investigating</strong> - We're currently seeing an issue in all regions where the IDE is failing to connect to hosted workspaces.
<br />
<br />Unable to access your workspace
<br />/mnt/shared/sbin/node
<br />ssh process died with exit code 255
<br />ssh: connect to host docker-gce-<region>-###-prod port #####: Connection timed out
<br />
<br />We're currently looking into this; please email support@c9.io if you have any questions.</p>tag:status.c9.io,2005:Incident/14471852017-10-27T16:29:13Z2017-10-27T16:29:39ZWorkspace restores are failing<p><small>Oct 27, 16:29 UTC</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Oct 27, 10:08 UTC</small><br><strong>Investigating</strong> - Our metrics indicate that a minimal number of workspace restores are failing. We're investigating the root cause now</p>tag:status.c9.io,2005:Incident/13925922017-10-02T19:02:02Z2017-10-02T19:02:02ZAPI latency is slow in USW<p><small>Oct 2, 19:02 UTC</small><br><strong>Resolved</strong> - API latency is back to normal</p><p><small>Oct 2, 18:57 UTC</small><br><strong>Monitoring</strong> - API latency is starting to lower</p><p><small>Oct 2, 18:51 UTC</small><br><strong>Identified</strong> - We've identified API latency to be slow in USW, causing timeouts and slow restores.</p><p><small>Oct 2, 18:44 UTC</small><br><strong>Investigating</strong> - Workspace loading and restores are very slow</p>tag:status.c9.io,2005:Incident/13731372017-09-27T14:49:27Z2017-09-27T14:49:27ZSupport chat inside of the IDE is currently disabled for premium customers<p><small>Sep 27, 14:49 UTC</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Sep 22, 13:33 UTC</small><br><strong>Monitoring</strong> - We have temporarily disabled support chat inside of the IDE for premium customers while we fix some issues with this feature.
<br />
<br />You can email support@c9.io with any support enquiries in the meantime.</p>tag:status.c9.io,2005:Incident/13837362017-09-27T10:57:00Z2017-09-27T14:23:04ZPartial outage on cs50.io<p><small>Sep 27, 10:57 UTC</small><br><strong>Resolved</strong> - IDE does not load on cs50.io</p>tag:status.c9.io,2005:Incident/13717482017-09-21T21:10:32Z2017-09-21T21:10:32ZWorkspaces inaccessible<p><small>Sep 21, 21:10 UTC</small><br><strong>Resolved</strong> - The issue was resolved. Any affected workspaces should be returning back to normal soon.</p><p><small>Sep 21, 20:18 UTC</small><br><strong>Investigating</strong> - We're seeing an issue with one of our hosts, which may cause customers to not access their workspaces.</p>tag:status.c9.io,2005:Incident/13684472017-09-20T12:56:56Z2017-09-20T12:56:56ZHigh latency in EU<p><small>Sep 20, 12:56 UTC</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Sep 20, 12:39 UTC</small><br><strong>Investigating</strong> - Decreased api latency and performance in the eu</p>tag:status.c9.io,2005:Incident/13682142017-09-20T10:38:48Z2017-09-20T10:38:48ZSome hosted workspaces inaccessible in Asia<p><small>Sep 20, 10:38 UTC</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Sep 20, 09:41 UTC</small><br><strong>Investigating</strong> - We are currently investigating this issue.</p>tag:status.c9.io,2005:Incident/13431852017-08-30T18:59:12Z2017-08-30T18:59:12ZHigh API Latencies in US<p><small>Aug 30, 18:59 UTC</small><br><strong>Resolved</strong> - Latencies have subsided. If you're still running into issues, please contact support@c9.io .</p><p><small>Aug 30, 18:37 UTC</small><br><strong>Investigating</strong> - We're currently seeing some higher-than-average API latencies in the US region. This is currently manifesting in additional lag and 500 errors on page load.</p>tag:status.c9.io,2005:Incident/13417582017-08-30T08:10:41Z2017-08-30T08:10:41ZSlow api response times<p><small>Aug 30, 08:10 UTC</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Aug 29, 16:31 UTC</small><br><strong>Investigating</strong> - We are currently experiencing slow api response times in USW and AE</p>tag:status.c9.io,2005:Incident/13416132017-08-29T14:59:23Z2017-08-29T14:59:23ZSlow api response times<p><small>Aug 29, 14:59 UTC</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Aug 29, 14:52 UTC</small><br><strong>Identified</strong> - We're currently seeing slow api response times. Creating and loading workspaces may fail, active workspaces should not be affected.</p>tag:status.c9.io,2005:Incident/13164872017-08-03T21:47:41Z2017-08-03T21:47:41ZElevated Error Levels for Cloud9 in US<p><small>Aug 3, 21:47 UTC</small><br><strong>Resolved</strong> - Error rates have dropped. Service should be working as usual. If you have any questions, please contact support@c9.io</p><p><small>Aug 3, 20:36 UTC</small><br><strong>Investigating</strong> - Cloud9 is running into high error levels in US. Workspaces, workspace previews, and dashboards may be affected.</p>tag:status.c9.io,2005:Incident/13147792017-08-02T15:33:34Z2017-08-02T15:33:34ZDegraded performance in US<p><small>Aug 2, 15:33 UTC</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Aug 2, 14:51 UTC</small><br><strong>Investigating</strong> - We are currently investigating this issue.</p>tag:status.c9.io,2005:Incident/13027962017-07-20T13:39:37Z2017-07-20T13:39:37ZHigh Latencies in EU<p><small>Jul 20, 13:39 UTC</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Jul 20, 13:23 UTC</small><br><strong>Investigating</strong> - Cloud9 is currently experiencing extremely high latencies in the EU, affecting all services. Other regions are unaffected.</p>tag:status.c9.io,2005:Incident/13007952017-07-19T17:15:05Z2017-07-19T17:15:05ZHigh Latencies in US<p><small>Jul 19, 17:15 UTC</small><br><strong>Resolved</strong> - Hosts have recovered and normal latencies have returned.</p><p><small>Jul 19, 15:49 UTC</small><br><strong>Investigating</strong> - Cloud9 is currently experiencing extremely high latencies in the US, affecting all services. Other regions are unaffected.</p>tag:status.c9.io,2005:Incident/12998772017-07-18T22:52:07Z2017-07-18T22:52:07ZCloud9 down in US<p><small>Jul 18, 22:52 UTC</small><br><strong>Resolved</strong> - US Latency is improving across the board. Pages should now load, albeit not at full speed.</p><p><small>Jul 18, 21:52 UTC</small><br><strong>Monitoring</strong> - US Latency is improving across the board. Pages should now load, albeit not at full speed.</p><p><small>Jul 18, 21:08 UTC</small><br><strong>Investigating</strong> - Cloud9 is currently experiencing extremely high latency in the US region, affecting all services. Other regions are unaffected.</p>tag:status.c9.io,2005:Incident/12912802017-07-11T21:30:41Z2017-07-11T21:30:41ZPassword Reset is failing<p><small>Jul 11, 21:30 UTC</small><br><strong>Resolved</strong> - Our email system has recovered successfully. This outage affected both password reset emails as well as password creation emails for new users. Both of these issues can be resolved by going to https://c9.io/auth/password/reset and requesting a password reset for your account email address. Please ensure that support@c9.io is not filtered to spam as this will contain your password reset email.
<br />
<br />If you did not need to create a new password during this time, you should be unaffected.</p><p><small>Jul 11, 18:17 UTC</small><br><strong>Investigating</strong> - We're experiencing issues with our Password Reset functionality. We are currently looking into this.</p>tag:status.c9.io,2005:Incident/12900732017-07-10T17:35:52Z2017-07-10T17:35:52ZPartial outage in AE. Issues when accessing workspaces on one host<p><small>Jul 10, 17:35 UTC</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Jul 10, 17:30 UTC</small><br><strong>Investigating</strong> - We are investigating partial outage on one of the hosts in AE. We will come back with more details.</p>tag:status.c9.io,2005:Incident/12898122017-07-10T16:40:02Z2017-07-10T16:40:02ZOne host in Asia is experiencing issues<p><small>Jul 10, 16:40 UTC</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Jul 10, 14:50 UTC</small><br><strong>Investigating</strong> - We are currently investigating this issue.</p>tag:status.c9.io,2005:Incident/12872652017-07-07T21:12:54Z2017-07-07T21:12:54ZWorkspace connection issues in Asia<p><small>Jul 7, 21:12 UTC</small><br><strong>Resolved</strong> - All workspaces can now be created and accessed as normal, at full speed, with a fully functional connection.</p><p><small>Jul 7, 17:39 UTC</small><br><strong>Investigating</strong> - Users are seeing workspace connection issues. We're currently looking into what's causing this.</p>tag:status.c9.io,2005:Incident/12683292017-06-16T23:04:08Z2017-06-16T23:04:08ZIssues when cloning hibernated workspaces<p><small>Jun 16, 23:04 UTC</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Jun 13, 15:00 UTC</small><br><strong>Investigating</strong> - We are seeing issues with cloning workspaces that are already hibernated. We are actively working to fix this as soon as possible, however in the meantime you can open the hibernated workspace before attempting to clone it as a workaround.</p>tag:status.c9.io,2005:Incident/12614942017-06-09T09:44:13Z2017-06-09T09:44:13ZMaintenance<p><small>Jun 9, 09:44 UTC</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Jun 9, 07:40 UTC</small><br><strong>Update</strong> - We're performing maintenance on some of our systems. In the meantime, customers in the EU region may speed or connection issues for your hosted workspaces.</p><p><small>Jun 8, 11:39 UTC</small><br><strong>Identified</strong> - We're performing maintenance on some of our systems. In the meantime, you may see speed or connection issues for your account or workspaces.</p>tag:status.c9.io,2005:Incident/12541402017-05-30T07:09:44Z2017-05-30T07:09:44ZWorkspace connection issues<p><small>May 30, 07:09 UTC</small><br><strong>Resolved</strong> - All workspaces can now be created and accessed as normal, at full speed, with a fully functional connection.</p><p><small>May 30, 06:31 UTC</small><br><strong>Investigating</strong> - Users are seeing workspace connection issues. We're currently looking into what's causing this.</p>tag:status.c9.io,2005:Incident/12414952017-05-15T10:18:33Z2017-05-15T10:18:33ZWorkspace opening issues in Asia<p><small>May 15, 10:18 UTC</small><br><strong>Resolved</strong> - Everything should be back to normal.</p><p><small>May 15, 08:10 UTC</small><br><strong>Investigating</strong> - There are currently issues affecting a small number of workspaces in our Asia region</p>