Traffic patterns and service have normalized to regular levels observed prior to the maintenance window. We will conduct an incident analysis and write up a blog post for this next week. Thank you for your patience and I'm sorry that this impacted your workflows.

Posted Apr 16, 2020 - 19:24 PDT

Update

We've implemented a short term workaround to restore service. We're monitoring the service.

Posted Apr 16, 2020 - 19:13 PDT

Update

We're isolating the issue with authz service's database queries that is taking an abnormally long time to complete.

Posted Apr 16, 2020 - 18:58 PDT

Update

We're investigating an unexpected elevation in fetches by the authz service from the database.

Posted Apr 16, 2020 - 17:00 PDT

Investigating

After upgrading PostgreSQL we are seeing database connections and CPU spikes. We're resizing the database to get more system resources and will provide additional updates as we have them.