Resolved -
We have worked through the accumulated backlog for private macOS builds which are now performing normally. We are continuing to monitor this closely. Thanks for your patience!
Sep 22, 10:58 UTC

Monitoring -
In an effort to stabilize our macOS infrastructure, yesterday at 17:00 UTC we also reduced the capacity available for macOS private repositories. We've now been able to address this and since 08:40 UTC, macOS Private builds are now running as expected. Thank you for your understanding.
Sep 22, 09:42 UTC

Resolved -
We've worked through the accumulated backlog for public macOS builds which are now performing normally. We are continuing to monitor this closely. Thanks for your patience!
Sep 22, 10:03 UTC

Update -
We have worked through most of the backlog over the last few hours. We continue to investigate the root cause for instability and we’ll post an update as soon as we can.
Sep 21, 09:00 UTC

Update -
We've completed the first round of job cancellations and are seeing some improvements. We're currently evaluating other changes to help reduce the backlog and improve the wait time for jobs starting. We'll provide more updates as we learn more. Thank you for your patience.
Sep 20, 15:44 UTC

Update -
In an effort to clear up load on macOS servers, we are cancelling builds that have been waiting to start for more than 6 hours. This process will start at 16:15 CEST and is expected to take approximately 2 hours to complete.

We will also be rolling back to a previous version of the worker to determine if recent changes have contributed to these issues.
Sep 20, 13:47 UTC

Update -
We continue to battle with stability issues on the MacOS platform, which is the root cause of severe wait times for public repository builds. We will keep you posted about any updates. We apologize for the delays this is causing.
Sep 20, 10:49 UTC

Update -
We have not made any significant gains in understanding the source of instability, but changes to available capacity distribution are helping to reduce the impact of disconnections.
Sep 20, 01:33 UTC

Investigating -
We’re investigating a higher-than-normal AMQP timeout errors affecting the throughput of our builds. This is affecting public OSX / mac OS jobs most at the moment, as this is the highest demand queue.
Sep 19, 10:50 UTC

Monitoring -
We have incurred a backlog on our sudo-enabled private repository queue while performing a graceful restart. We expect the backlog to clear once the full capacity is back online after finishing the longest-running jobs.
Sep 21, 15:28 UTC

Monitoring -
We can see that the backlog for sudo-enabled builds running on GCE has cleared. We are continuing to roll out the fix to our other infrastructures.
Sep 18, 20:56 UTC

Update -
We've identified that a new backend change was having a unexpected negative impact on communications with a message queue and was leading to an increased backlog. We've test out a configuration change to disable this new change and it's having the positive impact we expected, so we're continue to roll it out to all parts of our infrastructure. We'll provide updates as this rollout progress.
Sep 18, 20:03 UTC

Identified -
AWS is reporting issues with S3 in us-east-1, 11:58 AM PDT We are investigating increased error rates for Amazon S3 requests in the US-EAST-1 Region. WE can confirm we're seeing these issues as well. While S3 is unstable you'll see errors from build caching/artifacts activities and may have trouble accessing older build logs, which are stored in S3 long term. We will provide updates as we learn more.
Sep 14, 19:08 UTC

Resolved -
A tiny backlog remains for private Mac builds but it should be cleared in the next 30 minutes. Hence, we are resolving this incident for now. Thank you for your enduring patience!
Sep 14, 20:08 UTC

Update -
The backlog for container-based (i.e. sudo: false) Linux builds has cleared. A backlog remains for Mac builds and we will update you when it's cleared. Thank you!
Sep 14, 18:54 UTC

Update -
We are happy to report that the backlog has cleared for sudo-enabled Linux builds.

Monitoring -
We are sorry to inform you that the previous incident (https://www.traviscistatus.com/incidents/4gy46v0t3vrq), although it's fixed, resulted in a backlog for private builds. Hence you might experience some delays with your builds. Sorry for the inconvenience.

We are monitoring things closely and we will update with the state of the backlog on our different infrastructures in a timely manner.