We are currently investigating a delay in outbound email notifications for some customers.

10:37 AM PDT:

We’re still investigating the delays on outbound email notifications affecting some customers

10:56 AM PDT:

We’ve identified issue affecting outbound email notification delays and are working to resolve.

11:16 AM PDT:

We are still working to resolve the issue impacting some customer's email notification delays. Next update in 30 minutes

11:47 AM PDT:

We are still working to resolve the issue impacting some customer's email notification delays.

12:23 PM PDT:

We are still working to resolve the issue impacting some customer's email notification delays.

12:57 PM PDT:

We are working to resolve the backlog impacting some customer's email notification delays. Will update when we have a significant change

2:00 PM PDT:

Our Devs are still looking at the issue

3:00 PM PDT:

Our Devs are still looking at the issue

4:10 PM PDT:

All-clear. There are no more email delays up to this point.

POST-MORTEM

The root-cause of this issue was found to be an email loop that was sending a large amount of automated emails at a faster rate than what our systems could process. This in-turn affected other customers outbound email that were located in that same datacenter. The email loop itself was not identified as the cause of this issue until we had another observed email delay later in the day.

For this specific outage, a resolution was found by applying extra resources to the email queues in order to process the email at a faster rate than the jobs were coming in. While this process was occurring, our developers were continuing to investigate the increased load on the email processors as the initial belief was that the cause was from a product release earlier in the day.

Please subscribe to this article for regular updates until the issue is resolved. If you aren't subscribed to our Twitter feed, we encourage you to do so in order to get the most current information about any service issues. We also record all site outages on our system status page where you can see the past 12 months of service uptime. If you have questions about this issue, please open a ticket with us by sending a note to support@zendesk.com.