Service Incident - October 18th, 2019 - Sell Services Inaccessible

SUMMARY

On October 18, 2019 from 05:15 UTC to 06:11 UTC Zendesk Sell customers were unable to load dashboards, leads, deals and contacts and the API was not working.

06:36 UTC | 23:36 PT We're happy to report that the issues impacting Zendesk Sell have been resolved. Thank you for your patience.

06:20 UTC | 23:20 PT We are currently investigating connectivity and performance issues with Zendesk Sell and associated APIs. More info to come shortly.

Root Cause Analysis

This incident was caused by issues in the underlying storage subsystems of a specific data volume of the database, resulting in the MySQL engine to become unresponsive. Services that depend directly on the database cluster failed to process requests, which prevented the API from working. Additionally, some front end components of the app failed to load due to dependencies on affected services.

Resolution

A database failover was performed to resolve the issue, along with a manual redeploy of services to ensure front end components were able to load properly.

Remediation Items

Make improvements to the database architecture to minimize risk of recurrence

Investigate further automation of recovery process to reduce impact duration

FOR MORE INFORMATION

For current system status information about your Zendesk, check out our system status page. During an incident, you can also receive status updates by following @ZendeskOps on Twitter. The summary of our post-mortem investigation is usually posted here a few days after the incident has ended. If you have additional questions about this incident, please log a ticket with us.