18:46 UTC | 11:46 PT We have now resolved the issue with email delays in Pod 6. Please let us know if you're seeing continued problems.

18:05 UTC | 11:05 PT We’ve isolated the root cause of the email delays and the issue is improving. We will continue to monitor and provide an update shortly.

17:40 UTC | 10:40 PT We are seeing improvement in inbound mail delays for our Pod 6 customers. Updates to follow as we reach full resolution.

17:20 UTC | 10:20 PT We are investigating inbound email delays for some customers on Pod 6. More information to follow.

POST-MORTEM

During this incident, ticket creation via E-mail was delayed for up to 40 minutes in Pod 6. An email loop caused an abnormally large email chain that delayed incoming tickets by email for customers on Pod 6. To resolve the incident, we emptied the mail queues into a temporary folder. We then introduced the mail in the temporary folder back into the mail processing queue at a slower rate while monitoring its performance watching for timeouts and queue backups. To prevent this from happening in the future, we are deploying code to prevent mail loops on our email servers.

FOR MORE INFORMATION

For current system status information about your Zendesk, check out our system status page. During an incident, you can also receive status updates by following @ZendeskOps on Twitter. The summary of our post-mortem investigation is usually posted here a few days after the incident has ended. If you have additional questions about this incident, please log a ticket with us.