SUMMARY

17:57 UTC | 09:57 PT Inbound email delivery times are returning to normal in Pod 6. We are continuing to monitor the situation and will update shortly.

17:23 UTC | 09:23 PT We are still investigating delays with inbound email in Pod 6. Will update in 30 minutes.

16:49 UTC | 08:49 PT We are currently experiencing delays with inbound email in pod 6. We are investigating the issue and will update shortly.

POST-MORTEM

During this incident, customers on Pod 6 experienced inbound email delays for up to 50 minutes, causing ticket creation and ticket updates to also be delayed. This incident was caused by two simultaneous activities by spammers, resulting in email queue backups.

First, spammers sent Zendesk a large quantity of spam messages, which is nothing new.

Second, spammers created several accounts using fake email addresses in two non-Zendesk applications. These non-Zendesk applications sent messages to the fake email addresses. These messages included Zendesk Support addresses as the From: address, resulting in bounce-back emails to bog down our queue.

To resolve the issue, we put emails from the two non-Zendesk applications on hold until we could identify and block both the spam and the bounce notification emails. Email queues returned to normal after the offending messages were blocked and removed. Short-term and long-term solutions have been discussed and are in progress to minimize this type of malicious activity from happening in the future.

FOR MORE INFORMATION

For current system status information about your Zendesk, check out our system status page. During an incident, you can also receive status updates by following @ZendeskOps on Twitter. The summary of our post-mortem investigation is usually posted here a few days after the incident has ended. If you have additional questions about this incident, please log a ticket with us.