Dates:

May 10th, 10:29 PM - May 10th, 2019 11:31 PM

What happened:

Some automated tests using Appium in our real device cloud would not start between 10:29 PM and 11:31 PM (PDT.)

Why it happened:

One of our users started an extremely high number of simultaneous tests, significantly surpassing their allotted concurrency level. Our concurrency limit enforcement mechanisms did not work as expected and all the tests were allowed to run at once. This overloaded our system and most other tests were not able to start.

How we fixed it:

We suspended the offending account.

What we are doing to prevent it from happening again:

We made changes to our system in order to be more aggressive about checking and enforcing concurrency limits.

Posted 4 months ago. May 20, 2019 - 02:36 PDT

Resolved

We have identified the root cause and have fully recovered. All services are fully operational.

Posted 4 months ago. May 10, 2019 - 23:31 PDT

Update

We are taking remedial action. Tests are still not starting.

Posted 4 months ago. May 10, 2019 - 23:05 PDT

Investigating

Automated tests are not starting in our US Real Device Cloud. We are investigating.