The replacement load balancer is performing as expected. We will use any information that stems from the root cause analysis with AWS to improve our practices going forward.

Posted 12 months ago. Nov 21, 2017 - 20:32 UTC

Monitoring

We will continue to work with AWS to pursue root cause analysis of the problematic load balancer, but in the meantime we have provisioned a replacement load balancer which is now in service. We are monitoring closely, but we expect this will resolve the issue at hand.

Posted 12 months ago. Nov 21, 2017 - 19:17 UTC

Identified

The root cause of the errors seems to be a single AWS Load Balancer listener getting into a state where it times out all connections. We believed that removing and re-adding this listener was enough to get the out of this failure mode, but the timeouts eventually returned. We have now removed the listener again and we are working with AWS Support to determine best next steps.

Posted 12 months ago. Nov 21, 2017 - 18:23 UTC

Monitoring

We believe we have identified and addressed the root cause of the application errors. We are monitoring the situation and will update if things change.

Posted 12 months ago. Nov 21, 2017 - 17:30 UTC

Investigating

We are still encountering application errors and we are working on establishing the root cause.

Posted 12 months ago. Nov 21, 2017 - 17:09 UTC

Monitoring

Between 15:15 UTC and 15:50 UTC, Terraform Enterprise had a high rate of application errors. We have taken steps to remediate the errors and the error levels have returned to normal. We are investigating the root cause.