tag:status.hashicorp.com,2005:/historyHashiCorp Services Status - Incident History2018-11-20T00:42:52ZHashiCorp Servicestag:status.hashicorp.com,2005:Incident/20225042018-11-07T19:56:51Z2018-11-07T19:56:51ZTerraform Enterprise is currently experiencing site issues<p><small>Nov <var data-var='date'> 7</var>, <var data-var='time'>19:56</var> UTC</small><br><strong>Resolved</strong> - The failure was contained and we will continue looking into the cause of the issue. Based on logs, we believe the issue started at 18:44 UTC and was resolved by 19:08 UTC.</p><p><small>Nov <var data-var='date'> 7</var>, <var data-var='time'>19:27</var> UTC</small><br><strong>Identified</strong> - Site and all functions should be operational. The issue has been identified and more resources were added while existing ones recovered. Currently reviewing cause.</p><p><small>Nov <var data-var='date'> 7</var>, <var data-var='time'>18:58</var> UTC</small><br><strong>Investigating</strong> - We are currently investigating this issue.</p>tag:status.hashicorp.com,2005:Incident/20168612018-11-06T04:40:17Z2018-11-06T04:40:17ZTerraform Enterprise Maintenance<p><small>Nov <var data-var='date'> 6</var>, <var data-var='time'>04:40</var> UTC</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Nov <var data-var='date'> 6</var>, <var data-var='time'>04:00</var> UTC</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Nov <var data-var='date'> 5</var>, <var data-var='time'>17:37</var> UTC</small><br><strong>Scheduled</strong> - Terraform Enterprise will undergo routine database maintenance. During this window there will be an elevated risk of service related problems. Users may experience connection resets as well as delayed or incomplete Terraform runs.</p>tag:status.hashicorp.com,2005:Incident/19595242018-10-10T02:17:20Z2018-10-10T02:17:20ZTerraform Enterprise Plan and Apply Network Instability<p><small>Oct <var data-var='date'>10</var>, <var data-var='time'>02:17</var> UTC</small><br><strong>Resolved</strong> - Upstream networking has been stabilized.</p><p><small>Oct <var data-var='date'> 9</var>, <var data-var='time'>22:55</var> UTC</small><br><strong>Update</strong> - We are continuing to work on a fix for this issue.</p><p><small>Oct <var data-var='date'> 9</var>, <var data-var='time'>20:18</var> UTC</small><br><strong>Identified</strong> - The network supporting Terraform Enterprise plan and apply runs is experiencing connectivity issues with an upstream provider. We are working with the provider to resolve these issues. Plan and apply runs may experience network disconnects when communicating with some remote resources.</p>tag:status.hashicorp.com,2005:Incident/19538652018-10-08T17:38:50Z2018-10-08T17:38:50ZTerraform Enterprise User Account Creation Temporarily Disabled<p><small>Oct <var data-var='date'> 8</var>, <var data-var='time'>17:38</var> UTC</small><br><strong>Resolved</strong> - This issue has been resolved, and Terraform Enterprise user account creation is now enabled.</p><p><small>Oct <var data-var='date'> 6</var>, <var data-var='time'>21:14</var> UTC</small><br><strong>Identified</strong> - As we work to address an underlying issue, we are temporarily disabling user account creation within Terraform Enterprise. We will follow up once user account creation is back online. Terraform Enterprise customers can contact support if you need to have a user account created in the meantime.</p>tag:status.hashicorp.com,2005:Incident/19336072018-09-26T23:36:30Z2018-09-26T23:36:30ZIssues Ingressing a Configuration Version and Queueing a Run<p><small>Sep <var data-var='date'>26</var>, <var data-var='time'>23:36</var> UTC</small><br><strong>Resolved</strong> - This incident has been resolved. All Terraform runs should be queuing and completed as expected.</p><p><small>Sep <var data-var='date'>26</var>, <var data-var='time'>22:01</var> UTC</small><br><strong>Monitoring</strong> - We have identified the source of the issue and are continuing to monitor. Terraform Runs appear to be queuing and completing as expected at this time.</p><p><small>Sep <var data-var='date'>26</var>, <var data-var='time'>20:42</var> UTC</small><br><strong>Investigating</strong> - We are currently investigating an issue occurring when ingressing a configuration version or queuing a Terraform Run that causes all new queued runs to display "Error Queuing Plan: Invalid Run Parameters".</p>tag:status.hashicorp.com,2005:Incident/19046032018-09-12T20:19:55Z2018-09-12T20:19:55ZApplication Errors Downloading Artifacts<p><small>Sep <var data-var='date'>12</var>, <var data-var='time'>20:19</var> UTC</small><br><strong>Resolved</strong> - This incident has been resolved and all components of the application are operational.</p><p><small>Sep <var data-var='date'>12</var>, <var data-var='time'>20:11</var> UTC</small><br><strong>Monitoring</strong> - We have identified the issue and deployed a fix. We are monitoring closely.</p><p><small>Sep <var data-var='date'>12</var>, <var data-var='time'>20:03</var> UTC</small><br><strong>Investigating</strong> - We are investigating an issue affecting our artifact service and affecting the majority of operations in the application.</p>tag:status.hashicorp.com,2005:Incident/19029462018-09-12T01:00:15Z2018-09-12T01:00:15ZTerraform Runs Marked as Needs Confirmation<p><small>Sep <var data-var='date'>12</var>, <var data-var='time'>01:00</var> UTC</small><br><strong>Resolved</strong> - We have confirmed that the fix deployed resolves the issue.</p><p><small>Sep <var data-var='date'>12</var>, <var data-var='time'>00:03</var> UTC</small><br><strong>Monitoring</strong> - We have deployed a fix for the issue and will continue to monitor it.</p><p><small>Sep <var data-var='date'>11</var>, <var data-var='time'>23:33</var> UTC</small><br><strong>Update</strong> - We have now resolved the issue with runs marked as "Planned" locking the workspace in which they are running. The user interface is still incorrectly marking some runs as "Needs Confirmation", and we are continuing to work on a fix for that issue.</p><p><small>Sep <var data-var='date'>11</var>, <var data-var='time'>22:16</var> UTC</small><br><strong>Update</strong> - We have identified the commit causing plans with no changes to be marked as “Needs Confirmation” in the UI. Additionally, runs marked as “Planned” may lock the workspace in which they’re running. If the workspace is locked, it is safe to unlock the workspace via the workspace settings page and proceed with any queued or pending plans.</p><p><small>Sep <var data-var='date'>11</var>, <var data-var='time'>21:58</var> UTC</small><br><strong>Identified</strong> - We have identified the commit causing plans with no changes to be marked as “Needs Confirmation” in the UI. Additionally, runs marked as “Planned” may lock the workspace in which they’re running. If the workspace is locked, it is safe to unlock the workspace and proceed with any queued or pending plans.</p><p><small>Sep <var data-var='date'>11</var>, <var data-var='time'>21:42</var> UTC</small><br><strong>Investigating</strong> - The user interface is incorrectly marking some runs as "Needs Confirmation". We are investigating.</p>tag:status.hashicorp.com,2005:Incident/18980332018-09-08T03:34:49Z2018-09-11T18:42:24ZWorkspaces Unavailable<p><small>Sep <var data-var='date'> 8</var>, <var data-var='time'>03:34</var> UTC</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Sep <var data-var='date'> 8</var>, <var data-var='time'>01:15</var> UTC</small><br><strong>Update</strong> - The issue has been fully resolved and all systems are normal.</p><p><small>Sep <var data-var='date'> 8</var>, <var data-var='time'>00:42</var> UTC</small><br><strong>Monitoring</strong> - System should operational now, though we are continuing to monitor and recover the system.</p><p><small>Sep <var data-var='date'> 8</var>, <var data-var='time'>00:36</var> UTC</small><br><strong>Update</strong> - The team has identified the cause of the issue and is still working to resolve this. We will continue to provide a status update as it changes. Thank you for your patience.</p><p><small>Sep <var data-var='date'> 8</var>, <var data-var='time'>00:20</var> UTC</small><br><strong>Identified</strong> - Issue has been identified and we are working to resolve this. Will provide and update in the next ten minutes.</p><p><small>Sep <var data-var='date'> 8</var>, <var data-var='time'>00:07</var> UTC</small><br><strong>Investigating</strong> - Alert triggered for the site and investigating the cause. More updates soon.</p>tag:status.hashicorp.com,2005:Incident/18756652018-08-23T20:59:27Z2018-08-23T20:59:27ZDelayed Terraform Runs<p><small>Aug <var data-var='date'>23</var>, <var data-var='time'>20:59</var> UTC</small><br><strong>Resolved</strong> - Terraform Run Queue and Plan times have stabilized and the system is processing work as expected.</p><p><small>Aug <var data-var='date'>23</var>, <var data-var='time'>20:53</var> UTC</small><br><strong>Monitoring</strong> - We have identified an issue with delayed Terraform runs. Queue and plan times are beginning to stabilize and we are continuing to monitor this incident.</p>tag:status.hashicorp.com,2005:Incident/18693242018-08-20T16:22:09Z2018-08-20T18:30:30ZSentinel Policy Errors<p><small>Aug <var data-var='date'>20</var>, <var data-var='time'>16:22</var> UTC</small><br><strong>Resolved</strong> - From 16:22 - 17:54 UTC some Sentinel policy evaluations failed with error. This was caused by a bug in a code update to Sentinel. The update was reverted and functionality was restored.</p>tag:status.hashicorp.com,2005:Incident/18375052018-07-31T05:00:45Z2018-07-31T05:00:45ZTerraform Enterprise Matinence<p><small>Jul <var data-var='date'>31</var>, <var data-var='time'>05:00</var> UTC</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Jul <var data-var='date'>31</var>, <var data-var='time'>04:00</var> UTC</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Jul <var data-var='date'>30</var>, <var data-var='time'>20:45</var> UTC</small><br><strong>Scheduled</strong> - Terraform Enterprise will undergo routine database maintenance. During this window there will be an elevated risk of service related problems. Users may experience connection resets as well as delayed or incomplete Terraform runs.</p>tag:status.hashicorp.com,2005:Incident/18374552018-07-30T23:18:25Z2018-07-30T23:18:44ZSporadic Timeouts Downloading Modules From Terraform Module Registry<p><small>Jul <var data-var='date'>30</var>, <var data-var='time'>23:18</var> UTC</small><br><strong>Resolved</strong> - Our CDN provider was unable to discover any clear root cause, but offered some ideas for updates we can make that will allow us to better diagnose issues like this going forward.<br /><br />The Terraform Module Registry has continued to operate normally, so we are closing out this incident.</p><p><small>Jul <var data-var='date'>30</var>, <var data-var='time'>20:53</var> UTC</small><br><strong>Monitoring</strong> - Between 17:30 and 18:50 UTC today, we observed a period of sporadic timeouts between our content distribution network (CDN) and the application layer of the Terraform Module Registry. Everything on the Terraform Module Registry has been functioning normally since 18:50 UTC.<br /><br />These timeouts were not reflected in any of our application layer monitoring, leading us to believe that the root cause lies somewhere between the CDN and the application. We are reaching out to our CDN provider to see if they can provide further diagnosis.<br /><br />This incident exposed that we were missing some alerting on the CDN metrics that showed these timeouts. We will be closing this gap soon so we can catch issues like this more quickly in the future.</p><p><small>Jul <var data-var='date'>30</var>, <var data-var='time'>20:05</var> UTC</small><br><strong>Investigating</strong> - We are currently investigating reports of sporadic timeouts returned from `terraform init` operations as Terraform modules are being downloaded from the Registry. We will follow up with our findings.</p>tag:status.hashicorp.com,2005:Incident/18261472018-07-23T20:51:25Z2018-07-23T20:51:25ZTFE Stuck and Slow Runs<p><small>Jul <var data-var='date'>23</var>, <var data-var='time'>20:51</var> UTC</small><br><strong>Resolved</strong> - Terraform Runs are processing successfully and queue wait times have returned<br />to normal.</p><p><small>Jul <var data-var='date'>23</var>, <var data-var='time'>19:42</var> UTC</small><br><strong>Update</strong> - We are continuing to closely monitor this issue as worker queues begin to clear up. Terraform run, plan, and queue times are improving.</p><p><small>Jul <var data-var='date'>23</var>, <var data-var='time'>19:41</var> UTC</small><br><strong>Monitoring</strong> - We are continuing to closely monitor this issue as worker queues begin to clear up. Terraform run, plan, and queue times are improving.</p><p><small>Jul <var data-var='date'>23</var>, <var data-var='time'>19:06</var> UTC</small><br><strong>Investigating</strong> - We are actively investigating an issue causing stuck and slow runs in Terraform Enterprise. We will follow up with more information.</p>tag:status.hashicorp.com,2005:Incident/18179602018-07-18T02:27:45Z2018-07-18T02:27:45ZTerraform Run Slowness and Errors<p><small>Jul <var data-var='date'>18</var>, <var data-var='time'>02:27</var> UTC</small><br><strong>Resolved</strong> - The upstream provider has resolved this issue.</p><p><small>Jul <var data-var='date'>18</var>, <var data-var='time'>00:16</var> UTC</small><br><strong>Update</strong> - Services are not currently affected but risk continues to be higher than normal. We continue to monitor run times and await final resolution from the provider.</p><p><small>Jul <var data-var='date'>17</var>, <var data-var='time'>21:37</var> UTC</small><br><strong>Update</strong> - Services are not currently affected but risk continues to be higher than normal. We continue to monitor run times and await final resolution from the provider.</p><p><small>Jul <var data-var='date'>17</var>, <var data-var='time'>19:29</var> UTC</small><br><strong>Update</strong> - Update: The upstream network issues have stopped causing Terraform slowness and errors; however, we continue to monitor performance and are keeping the status listed as "Degraded Performance" until the upstream provider declares the issue resolved.</p><p><small>Jul <var data-var='date'>17</var>, <var data-var='time'>18:25</var> UTC</small><br><strong>Monitoring</strong> - At 17:19 UTC today, Terraform plan and apply runs began experiencing above normal run times and failure rates. These problems are being caused by an upstream provider networking issue. <br /><br />We are monitoring the situation for further updates and evaluating remediation steps.</p>tag:status.hashicorp.com,2005:Incident/17927392018-06-29T02:37:47Z2018-06-29T02:37:47ZNew Organization Creation Broken<p><small>Jun <var data-var='date'>29</var>, <var data-var='time'>02:37</var> UTC</small><br><strong>Resolved</strong> - The site has been rolled back to a previous release, addressing the underlying issue. Thank you for your patience.</p><p><small>Jun <var data-var='date'>29</var>, <var data-var='time'>01:18</var> UTC</small><br><strong>Identified</strong> - Customers may experience the inability to create new organizations. The underlying problem is known and the fix is being authored now.</p>tag:status.hashicorp.com,2005:Incident/17925052018-06-28T23:04:51Z2018-06-28T23:04:51ZTerraform Enterprise: Delayed Plan and Apply Operations<p><small>Jun <var data-var='date'>28</var>, <var data-var='time'>23:04</var> UTC</small><br><strong>Resolved</strong> - We have rolled back the effected release and restored proper operations.</p><p><small>Jun <var data-var='date'>28</var>, <var data-var='time'>22:12</var> UTC</small><br><strong>Monitoring</strong> - We have identified the cause of the issue and deployed a fix. All Plan and Apply operations should now be completing successfully. It is possible that some jobs may need to be canceled and retried. We will continue monitoring closely.</p><p><small>Jun <var data-var='date'>28</var>, <var data-var='time'>21:55</var> UTC</small><br><strong>Investigating</strong> - We are investigating reports of delayed Plan and Apply operations within Terraform Enterprise.</p>tag:status.hashicorp.com,2005:Incident/17922202018-06-28T19:56:50Z2018-06-28T19:56:50ZRegistry Module Outage<p><small>Jun <var data-var='date'>28</var>, <var data-var='time'>19:56</var> UTC</small><br><strong>Resolved</strong> - We have identified the root cause and the fix to prevent it from occurring in the future has been scheduled.</p><p><small>Jun <var data-var='date'>28</var>, <var data-var='time'>19:56</var> UTC</small><br><strong>Update</strong> - We have identified the root cause and the fix to prevent it from occurring in the future has been scheduled.</p><p><small>Jun <var data-var='date'>28</var>, <var data-var='time'>19:01</var> UTC</small><br><strong>Monitoring</strong> - We have identified the problem and implemented a remediation - Terraform Registry module ingress work is catching up. Some modules will continue to experience high latency for ingress but should eventually complete. We will continue to monitor closely.</p><p><small>Jun <var data-var='date'>28</var>, <var data-var='time'>18:37</var> UTC</small><br><strong>Update</strong> - A service rollback has not solved the issue and Terraform Registry modules are still not ingressing. We are continuing to narrow down the root cause of the problem.</p><p><small>Jun <var data-var='date'>28</var>, <var data-var='time'>18:07</var> UTC</small><br><strong>Investigating</strong> - There is currently an issue ingressing registry modules, preventing new modules from being seen in the system.</p>tag:status.hashicorp.com,2005:Incident/17871382018-06-26T07:02:28Z2018-06-26T07:02:28ZIssues with gitlab.com backed VCS connections<p><small>Jun <var data-var='date'>26</var>, <var data-var='time'>07:02</var> UTC</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Jun <var data-var='date'>26</var>, <var data-var='time'>02:31</var> UTC</small><br><strong>Update</strong> - The fix has been deployed and verified.</p><p><small>Jun <var data-var='date'>25</var>, <var data-var='time'>23:05</var> UTC</small><br><strong>Update</strong> - Functionality for existing Gitlab VCS connections has been restored. We are working on a fix for new connections.</p><p><small>Jun <var data-var='date'>25</var>, <var data-var='time'>21:15</var> UTC</small><br><strong>Identified</strong> - We have identified an issue with our Gitlab.com integration. Our engineers are working towards a solution.</p>tag:status.hashicorp.com,2005:Incident/17771032018-06-19T17:56:41Z2018-06-19T17:56:41ZRegistry module issues<p><small>Jun <var data-var='date'>19</var>, <var data-var='time'>17:56</var> UTC</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Jun <var data-var='date'>19</var>, <var data-var='time'>17:24</var> UTC</small><br><strong>Monitoring</strong> - Issue had been adressed. We will be monitoring the situation.</p><p><small>Jun <var data-var='date'>19</var>, <var data-var='time'>15:26</var> UTC</small><br><strong>Update</strong> - We are continuing to work on a fix for this issue.</p><p><small>Jun <var data-var='date'>19</var>, <var data-var='time'>15:20</var> UTC</small><br><strong>Identified</strong> - Users may be seeing failures when applying terraform plans if modules are used. We have identified the cause and working on a fix</p>tag:status.hashicorp.com,2005:Incident/17497762018-05-31T19:43:38Z2018-05-31T19:43:38ZTerraform Runs are Delayed<p><small>May <var data-var='date'>31</var>, <var data-var='time'>19:43</var> UTC</small><br><strong>Resolved</strong> - Terraform Run queues are now back to normal levels. We will continue to monitor.</p><p><small>May <var data-var='date'>31</var>, <var data-var='time'>19:33</var> UTC</small><br><strong>Investigating</strong> - Terraform Runs (plans and applies) are currently delayed by several minutes. We are investigating the cause.</p>tag:status.hashicorp.com,2005:Incident/17197052018-05-07T17:54:24Z2018-05-07T17:54:24ZTerraform Sentinel Policy Checks<p><small>May <var data-var='date'> 7</var>, <var data-var='time'>17:54</var> UTC</small><br><strong>Resolved</strong> - Terraform Runs with Sentinel Policy Checks should be running as of 40 minutes ago. Ongoing monitoring and investigation into the cause will continue.</p><p><small>May <var data-var='date'> 7</var>, <var data-var='time'>16:33</var> UTC</small><br><strong>Identified</strong> - Issue has been identified, but still working on resolving.</p><p><small>May <var data-var='date'> 7</var>, <var data-var='time'>15:15</var> UTC</small><br><strong>Investigating</strong> - Terraform runs with Sentinel Policy Checks are stuck in a queued state. Currently investigating.</p>tag:status.hashicorp.com,2005:Incident/17157922018-05-03T21:45:56Z2018-05-03T21:45:56ZTerraform Enterprise Outage<p><small>May <var data-var='date'> 3</var>, <var data-var='time'>21:45</var> UTC</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>May <var data-var='date'> 3</var>, <var data-var='time'>21:19</var> UTC</small><br><strong>Monitoring</strong> - We have identified the cause of the outage and stabilized the system.</p><p><small>May <var data-var='date'> 3</var>, <var data-var='time'>21:00</var> UTC</small><br><strong>Update</strong> - We are continuing to investigate this issue.</p><p><small>May <var data-var='date'> 3</var>, <var data-var='time'>20:44</var> UTC</small><br><strong>Investigating</strong> - We have detected an outage in Terraform Enterprise and are working to resolve as quickly as possible.</p>tag:status.hashicorp.com,2005:Incident/16560532018-03-23T19:19:40Z2018-03-23T19:19:40ZDelays Processing Terraform Runs<p><small>Mar <var data-var='date'>23</var>, <var data-var='time'>19:19</var> UTC</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Mar <var data-var='date'>23</var>, <var data-var='time'>18:34</var> UTC</small><br><strong>Monitoring</strong> - Queue wait times have improved; we are continuing to monitor closely.</p><p><small>Mar <var data-var='date'>23</var>, <var data-var='time'>18:00</var> UTC</small><br><strong>Identified</strong> - We have identified an issue causing delays in processing Terraform Runs and and working towards addressing it.</p>tag:status.hashicorp.com,2005:Incident/16502052018-03-20T17:25:41Z2018-03-20T17:25:41ZPartially Degraded Service - AWS US-EAST-1<p><small>Mar <var data-var='date'>20</var>, <var data-var='time'>17:25</var> UTC</small><br><strong>Resolved</strong> - Amazon has resolved their incident and we are not seeing further issues on our end.</p><p><small>Mar <var data-var='date'>20</var>, <var data-var='time'>15:55</var> UTC</small><br><strong>Monitoring</strong> - Terraform or Packer Runs could be affected if they rely on resources hosted in AWS. We are tracking upstream issues affecting AWS APIs in the us-east-1 region. We will continue to monitor as the status changes.</p>tag:status.hashicorp.com,2005:Incident/16429332018-03-15T17:30:56Z2018-03-15T17:30:56ZElevated Error Rates for Terraform Enterprise Apply Operations<p><small>Mar <var data-var='date'>15</var>, <var data-var='time'>17:30</var> UTC</small><br><strong>Resolved</strong> - At this stage we are not seeing any more errors. The root cause has been identified. We will be monitoring the situation, but do not expect any further issues.</p><p><small>Mar <var data-var='date'>15</var>, <var data-var='time'>16:58</var> UTC</small><br><strong>Investigating</strong> - We are investigating reports of "invalid run parameters" error messages and inability to apply Plans in Terraform Enterprise.</p>