Please refer to this postmortem to view postmortem information for this incident.

Posted about 1 year ago. Oct 04, 2017 - 18:12 UTC

Resolved

We are fully recovered with no delays in the delivery of notifications or webhooks. At this time, we have taken all reasonable measures to ensure the stability of our system and are closing this incident. Once we have completed a thorough analysis and investigation, we will publish a full postmortem.

Posted about 1 year ago. Sep 21, 2017 - 03:28 UTC

Update

At this time, we have fully recovered and are not seeing delays in the delivery of notifications.

As per our previous update, we have successfully made the change/upgrade to our distributed data infrastructure.

We are now monitoring the system to ensure we have long-term stability. As always, we will send another update on any material changes.

Posted about 1 year ago. Sep 21, 2017 - 03:02 UTC

Update

We are now starting to make a change/upgrade to our distributed data infrastructure in order to improve both short-term and long-term stability. We believe this change will cause temporary delays in notifications. We have all hands on deck working and monitoring the situation, in order to minimize the window of delays in notifications.

Posted about 1 year ago. Sep 21, 2017 - 01:17 UTC

Update

At this time, we have temporarily recovered and are no longer seeing delays in delivery of notifications. That being said, we are working on making a change to our distributed data infrastructure to permanently remediate the issue. We believe that this change will cause additional temporary delays in the delivery of notifications. We will provide an additional update just before undertaking this change.

Posted about 1 year ago. Sep 20, 2017 - 23:37 UTC

Update

We continue to work to remediate the delay in notifications and webhooks.

Posted about 1 year ago. Sep 20, 2017 - 20:25 UTC

Update

We are still working at remediation of this issue but continue to see a delay in delivery of notifications and webhooks averaging about 30 minutes.

Posted about 1 year ago. Sep 20, 2017 - 18:49 UTC

Update

We are still experiencing delays in notification processing and delivery at an average of 30 minutes. At this time, we are exploring multiple paths to remedy and bring our systems back to an acceptable and expected state. The core of our issue resides in our distributed data infrastructure with respect to maintaining acceptable throughput and performance of data across regions. At this time we do not have an ETA for full recovery but rest assured we have around the clock coverage to mitigate this issue.

Posted about 1 year ago. Sep 20, 2017 - 17:39 UTC

Update

Notifications and webhooks are still delayed but flowing. We are continuing to investigate and monitor the situation.

Posted about 1 year ago. Sep 20, 2017 - 16:52 UTC

Update

We are continuing to investigate and monitor the situation. Notifications and webhooks are still delayed but flowing.

Posted about 1 year ago. Sep 20, 2017 - 16:19 UTC

Update

Notifications and webhooks are delayed and we are actively working to mitigate the issue.

Posted about 1 year ago. Sep 20, 2017 - 15:43 UTC

Update

Notifications are delayed and we are actively working to mitigate the issue. We are continuing to monitor and investigate the situation.

Posted about 1 year ago. Sep 20, 2017 - 15:08 UTC

Update

We are continuing to monitor and investigate the situation. Notifications are going through but delivery times are delayed and still fluctuating.

Posted about 1 year ago. Sep 20, 2017 - 14:35 UTC

Update

No significant changes at this time. Notifications are going through, although delivery times continue to fluctuate. We will continue to monitor and investigate the situation.

Posted about 1 year ago. Sep 20, 2017 - 13:57 UTC

Update

Notifications are going through, although delivery times continue to fluctuate. We will continue to monitor and investigate the situation.

Posted about 1 year ago. Sep 20, 2017 - 13:17 UTC

Update

Notification delivery times are fluctuating. We’re still monitoring and investigating the situation.

Posted about 1 year ago. Sep 20, 2017 - 12:36 UTC

Monitoring

Notifications are going out in a timely manner. We are still monitoring the situation. All other systems are functional.

Posted about 1 year ago. Sep 20, 2017 - 12:01 UTC

Investigating

We are currently experiencing delays in notifications. Our engineering team is aware and working to restore notification delivery to full performance.