tag:status.pusher.com,2005:/historyPusher Status - Incident History2017-08-18T02:50:12ZPushertag:status.pusher.com,2005:Incident/13284052017-08-15T10:11:24Z2017-08-15T10:11:25Zdashboard.pusher.com unreachable<p><small>Aug 15, 10:11 UTC</small><br><strong>Resolved</strong> - We are confident that the Dashboard is back to full functionality. SSL issues that were reported were also investigated and fixed.</p><p><small>Aug 15, 08:04 UTC</small><br><strong>Monitoring</strong> - We've implemented a fix and are currently monitoring it. Live stats functionality has been restored.</p><p><small>Aug 15, 07:28 UTC</small><br><strong>Identified</strong> - We have identified the issue and a temporary fix is in place that will allow the Dashboard to be used. However, real-time stats will not be available till we've resolved the issue entirely. Core Dashboard functions should be unaffected.</p><p><small>Aug 15, 07:03 UTC</small><br><strong>Investigating</strong> - Users may experience issues accessing their dashboard. Investigation ongoing.</p>tag:status.pusher.com,2005:Incident/13146382017-08-02T17:10:25Z2017-08-02T17:10:26ZDashboard stats incomplete<p><small>Aug 2, 17:10 UTC</small><br><strong>Resolved</strong> - Stats should now be displaying correctly again in the dashboard.</p><p><small>Aug 2, 13:15 UTC</small><br><strong>Update</strong> - Pages on the dashboard that show stats about apps or accounts might time out for a short period starting from now.
<br />
<br />We will update the incident when the fix is complete.</p><p><small>Aug 2, 13:00 UTC</small><br><strong>Identified</strong> - We have identified the issue and are working on a fix now.
<br />
<br />There may be a brief period where pages on the dashboard that show stats about apps or accounts are unavailable but we will update the incident before this starts.</p><p><small>Aug 2, 12:52 UTC</small><br><strong>Investigating</strong> - The stats for apps and accounts showing on the dashboard are incomplete since 09:50 UTC today.
<br />
<br />We are investigating the issue.</p>tag:status.pusher.com,2005:Incident/12846852017-07-05T16:51:27Z2017-07-05T16:51:27ZPush Notification Issue<p><small>Jul 5, 16:51 UTC</small><br><strong>Resolved</strong> - We've identified and implemented necessary code optimizations in the way we deal with invalid requests so that it doesn't affect the rest of the service in the future. The fix has been deployed so the issue is now marked as resolved.</p><p><small>Jul 5, 14:09 UTC</small><br><strong>Monitoring</strong> - The service has stabilised and is performing normally. We will be monitoring and investigate more closely what happen.</p><p><small>Jul 5, 13:25 UTC</small><br><strong>Investigating</strong> - We are currently experiencing some issues with the Push Notifications service. We are investigating the cause and attempting to resolve the issue.</p>tag:status.pusher.com,2005:Incident/12785692017-06-28T15:07:09Z2017-06-28T15:07:10ZDatadog integration outage<p><small>Jun 28, 15:07 UTC</small><br><strong>Resolved</strong> - Datadog has been caught up with historic stats data and is now running at real-time. Closing the issue as the service is back to normal operation. We will be investigating further to set up additional tooling to help avoid any future stats integrations outages.</p><p><small>Jun 28, 13:22 UTC</small><br><strong>Monitoring</strong> - The issue has been fixed and historical stats data is now being uploaded to Datadog. We expect Datadog to be up to date and back to realtime stats monitoring within the next hour and a half (15:00 UTC). We will continue monitoring the situation and update this status page when normal operation has resumed.</p><p><small>Jun 28, 09:27 UTC</small><br><strong>Investigating</strong> - The component responsible for triggering the Datadog requests has been inactive since this morning around 02:00 UTC. We are investigating the issue now, once the system is fixed we expect to catch up Datadog with all historic data.</p>tag:status.pusher.com,2005:Incident/12649632017-06-13T19:36:48Z2017-06-13T19:36:49ZDatadog intergration outage<p><small>Jun 13, 19:36 UTC</small><br><strong>Resolved</strong> - Datadog has been caught up with historic stats data and is now running at real-time. Closing the issue as the problem is resolved. We will be investigating further and setting up further precautions to help avoid any future stats integrations outages.</p><p><small>Jun 13, 18:13 UTC</small><br><strong>Monitoring</strong> - The issue has been fixed and historical stats data is now being uploaded to Datadog. We expect Datadog to be up to date and back to realtime stats monitoring within the next hour (19:30 UTC). We will continue monitoring the situation and update this status page when normal operation has resumed.</p><p><small>Jun 13, 16:20 UTC</small><br><strong>Update</strong> - We have traced the problem to internal SSL connection failures. We are still investigating the cause of these failures. Stats are not lost and we expect Datadog to catch up when these internal connections are restored.</p><p><small>Jun 13, 10:34 UTC</small><br><strong>Investigating</strong> - The component responsible for triggering the Datadog requests has been inactive since this morning around 01:15 UTC. We are investigating the issue now, once the system is fixed we expect to catch up Datadog with all historic data.</p>tag:status.pusher.com,2005:Incident/12573332017-06-02T15:18:00Z2017-06-02T15:21:40ZStats Integrations<p><small>Jun 2, 15:18 UTC</small><br><strong>Resolved</strong> - Today at 13:45 until 15:05 UTC we had a partial outage in our third party stats integrations due to an internal change on our stats pipeline. The stats for that time period will not be forwarded to Datadog, but they are still available on our dashboard.
<br />
<br />The problem is resolved and we will be investigating further and setting up alarms to catch this kind of service interruption sooner.</p>tag:status.pusher.com,2005:Incident/11900972017-04-07T12:39:32Z2017-04-07T12:39:32ZRequest timeouts to the Push Notifications API<p><small>Apr 7, 12:39 UTC</small><br><strong>Resolved</strong> - Everything seems to be operational now. We're monitoring the situation closely.</p><p><small>Apr 7, 09:24 UTC</small><br><strong>Investigating</strong> - Some customers are experiencing timeouts when trying to make requests to the Push Notifications API. We're currently investigating and will update shortly.</p>tag:status.pusher.com,2005:Incident/11444362017-03-02T01:54:34Z2017-03-02T01:54:35ZDegraded Webhook Performance<p><small>Mar 2, 01:54 UTC</small><br><strong>Resolved</strong> - The problem causing an elevated error rate for webhook delivery has been resolved.</p><p><small>Mar 2, 01:37 UTC</small><br><strong>Monitoring</strong> - The problem with webhook delivery has been resolved and we are now monitoring the situation.</p><p><small>Mar 2, 01:17 UTC</small><br><strong>Investigating</strong> - There is currently an elevated error rate for webhook delivery. We are investigating the issue.</p>tag:status.pusher.com,2005:Incident/11277482017-02-17T19:17:59Z2017-02-17T19:17:59ZDatadog Integration Stats being forwarded with some delay<p><small>Feb 17, 19:17 UTC</small><br><strong>Resolved</strong> - The historical data was propagated to Datadog.
<br />
<br />The system is now sending stats in realtime.</p><p><small>Feb 17, 18:18 UTC</small><br><strong>Monitoring</strong> - The component responsible for triggering the Datadog requests was inactive since yesterday around 18:00 UTC.
<br />
<br />We have identified and fixed the underlying issue. An alarm was added to our monitoring system to catch this type of issue earlier.
<br />
<br />Stats are now being caught up to on and should reach realtime status in a couple of hours. We'll be monitoring this issue and we will update as soon all the historical metrics have reached Datadog.</p>tag:status.pusher.com,2005:Incident/11214382017-02-15T10:18:19Z2017-02-15T10:18:20ZTimeouts on Push Notification Service<p><small>Feb 15, 10:18 UTC</small><br><strong>Resolved</strong> - Timeouts have subsided and the service has returned to acceptable performance. We also tweaked some parameters that will allow the service to deal with increased loads. We also deployed a new version with better logging and metrics to allow us to react more quickly and devise a fix, if future incidents do occur.</p><p><small>Feb 14, 11:32 UTC</small><br><strong>Investigating</strong> - At 09:50 UTC the Push Notifications service was restarted and returned to acceptable performance. We are deploying additional metrics to understand the issue and determine a fix.</p><p><small>Feb 13, 14:33 UTC</small><br><strong>Identified</strong> - The problem has been narrowed down to a load issue caused when a very large batch of APNS publishes are requested within a small timeframe. We are improving our metric collection to better understand the issue.</p><p><small>Feb 13, 11:07 UTC</small><br><strong>Investigating</strong> - There appear to be intermittent request timeouts on our Push Notifications Service. We're actively investigating the cause.</p>tag:status.pusher.com,2005:Incident/11013252017-01-24T17:40:53Z2017-01-24T17:40:53ZTimeouts on Push Notification Service<p><small>Jan 24, 17:40 UTC</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Jan 24, 16:42 UTC</small><br><strong>Monitoring</strong> - A fix is in place and we're currently monitoring the system.</p><p><small>Jan 24, 13:49 UTC</small><br><strong>Investigating</strong> - There appear to be intermittent request timeouts on our Push Notifications Service. We're actively investigating the cause.</p>tag:status.pusher.com,2005:Incident/10233772016-12-15T09:07:30Z2016-12-15T09:07:30ZErrors reaching EU cluster<p><small>Dec 15, 09:07 UTC</small><br><strong>Resolved</strong> - This issue was completely resolved as of 03:17 when our upstream provider resolved their problems with DNS resolution.</p><p><small>Dec 15, 02:11 UTC</small><br><strong>Monitoring</strong> - Mitigations appear to be successful. Some percentage of websocket clients may continue to experience difficulty connecting while their DNS caches hold reference to the affected resources.</p><p><small>Dec 15, 01:43 UTC</small><br><strong>Identified</strong> - One of the EU availability zones appears to be suffering network disruptions, which are degrading service quality without being severe enough to have the resources automatically pulled from service. Mitigations are being put in place.</p><p><small>Dec 15, 01:34 UTC</small><br><strong>Investigating</strong> - We are investigating networking issues affecting access to our EU cluster</p>tag:status.pusher.com,2005:Incident/9904462016-12-12T10:21:37Z2016-12-12T10:21:37ZDashboard stats incomplete<p><small>Dec 12, 10:21 UTC</small><br><strong>Resolved</strong> - We have made changes to our storage layer, and having monitored for over a week since the changes, we are happy that the underlying issue has been resolved.</p><p><small>Nov 17, 18:26 UTC</small><br><strong>Monitoring</strong> - Data is moving correctly through to the dashboard again, though we are not satisfied that we have addressed the root cause yet.
<br />
<br />Unfortunately the data for the period 2016-11-17 15:00 - 16:05 UTC was lost. This may result in historic daily totals appearing short of the actual values, while limits are still being enforced based on accurate data.
<br />
<br />We continue to work to address the underlying performance problems which have prevented the timely presentation of data.</p><p><small>Nov 16, 18:00 UTC</small><br><strong>Investigating</strong> - The dashboard historic stats are suffering another backend failure. We are extremely frustrated by this turn of events and are working to make improvements. However, it seems likely that the historic stats pages will not receive new data for at least the next 24 hours.
<br />
<br />The data is still being collected, but it not making it to the presentation layer.
<br />
<br />Limits are still being enforced based on accurate data.
<br />
<br />Real-time stats showing usage during the last 30 minutes remain available on the account and app overview pages, so you should not be in the dark about the current behaviour of your Pusher apps.
<br />
<br />We apologise for this interruption to service.</p>tag:status.pusher.com,2005:Incident/10098642016-12-05T10:43:05Z2016-12-05T10:43:05ZDashboard maintenance period<p><small>Dec 5, 10:43 UTC</small><br><strong>Resolved</strong> - This maintenance is now complete.</p><p><small>Dec 5, 10:30 UTC</small><br><strong>Identified</strong> - Within the next hour we must take the dashboard (https://dashboard.pusher.com/) down for maintenance. We expect this to take no longer than 30 minutes.</p>tag:status.pusher.com,2005:Incident/9891522016-11-16T16:10:51Z2016-11-16T16:10:51ZDashboard stats incomplete<p><small>Nov 16, 16:10 UTC</small><br><strong>Resolved</strong> - All stats for the missing periods have been regenerated. Dashboard graphs are now showing correct and up-to-date stats for all time periods.</p><p><small>Nov 16, 10:37 UTC</small><br><strong>Update</strong> - We are continuing to work on resolving this issue. In the meantime, historical stats will remain incomplete.</p><p><small>Nov 15, 18:15 UTC</small><br><strong>Monitoring</strong> - We have identified the root cause of the issue.
<br />
<br />On the main (us-east-1) cluster, historical stats in the dashboard will not be visible until tomorrow morning UTC. We have not lost the data, and we will repopulate the graphs once the underlying issue is resolved.
<br />
<br />Limits are still being applied to accurate data, but the graphs in the limits email may also be inaccurate.</p><p><small>Nov 15, 16:29 UTC</small><br><strong>Identified</strong> - A problem pushing stats to our dashboard has resulted in graphs with partial data being displayed. The data has not been lost and the graphs will be re-generated once all data has been delivered to the dashboard.</p>tag:status.pusher.com,2005:Incident/9900712016-11-16T11:05:36Z2016-11-16T11:05:37ZDashboard maintenance scheduled within an hour<p><small>Nov 16, 11:05 UTC</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Nov 16, 10:55 UTC</small><br><strong>Monitoring</strong> - The dashboard is now in maintenance mode. We expect this to take no longer than one hour.</p><p><small>Nov 16, 10:31 UTC</small><br><strong>Identified</strong> - Within the next hour we must take the dashboard (https://dashboard.pusher.com/) down for maintenance. We expect this to take no longer than one hour.</p>tag:status.pusher.com,2005:Incident/9868752016-11-12T12:34:15Z2016-11-14T13:44:45ZExpired HTTP API certificate on the EU cluster<p><small>Nov 12, 12:34 UTC</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Nov 12, 12:33 UTC</small><br><strong>Investigating</strong> - This issue has been resolved.</p>tag:status.pusher.com,2005:Incident/9571182016-10-22T11:38:41Z2016-10-22T11:38:41ZDashboard Unavailable<p><small>Oct 22, 11:38 UTC</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Oct 21, 19:33 UTC</small><br><strong>Monitoring</strong> - We are currently monitoring an availability event with an upstream DNS provider which is causing intermittent availability of the Pusher Dashboard.</p>tag:status.pusher.com,2005:Incident/9554402016-10-21T11:11:55Z2016-10-21T11:11:55ZDashboard stats incomplete<p><small>Oct 21, 11:11 UTC</small><br><strong>Resolved</strong> - The mis-reported period has been regenerated. Dashboard graphs are now showing the correct values.</p><p><small>Oct 21, 09:53 UTC</small><br><strong>Identified</strong> - The issue has been identified and a fix is being implemented.</p><p><small>Oct 21, 09:53 UTC</small><br><strong>Investigating</strong> - A problem pushing stats to our dashboard has resulted in graphs with partial data being displayed. The data is not believed to have been lost and the graphs will be re-generated once all data has been delivered to the dashboard.</p>tag:status.pusher.com,2005:Incident/9247532016-10-01T08:59:08Z2016-10-01T08:59:09ZPush notifications service outage<p><small>Oct 1, 08:59 UTC</small><br><strong>Resolved</strong> - The push notifications service has been restored to normal operation as of 09:57 (GMT+01:00).</p><p><small>Oct 1, 08:27 UTC</small><br><strong>Investigating</strong> - Since 04:07AM (GMT+01:00) the push notifications service has been unable to connect to its database after a failover.
<br />
<br />We're currently investigating why it hasn't been able to continue as expected after the failover.</p>tag:status.pusher.com,2005:Incident/8698022016-09-26T13:14:25Z2016-09-26T13:15:58ZConnectivity Issues with AWS us-east-1<p><small>Sep 26, 13:14 UTC</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Sep 26, 12:21 UTC</small><br><strong>Update</strong> - Networking conditions appear to have cleared. We will continue to monitor the situation.</p><p><small>Sep 26, 12:12 UTC</small><br><strong>Monitoring</strong> - Although not formally acknowledged by AWS at this time, we are seeing reports from other users confirming that this is a problem at AWS.
<br />
<br />All clusters in us-east-1 (including MT1, our main multitenant cluster) appear to be reachable for customers and users in North America. Customers and users from the rest of the world are suffering degraded network connectivity to these clusters.
<br />
<br />Given that the affected clusters are still reachable by the overwhelming majority of their users, we do not intend to migrate them to an alternative AWS region at this time.
<br />
<br />We are monitoring the situation as it develops.</p><p><small>Sep 26, 12:04 UTC</small><br><strong>Investigating</strong> - We are currently investigating some connectivity issues with the us-east-1 AWS region. Specifically, network connectivity outside the US appears to be problematic.</p>tag:status.pusher.com,2005:Incident/8142722016-09-20T11:42:17Z2016-09-20T11:42:59ZDegraded performance on EU cluster<p><small>Sep 20, 11:42 UTC</small><br><strong>Resolved</strong> - Performance on the EU cluster has been restored.</p><p><small>Sep 20, 11:07 UTC</small><br><strong>Investigating</strong> - We are investigating degraded performance on EU cluster since 1030 UTC. Will update when we know more. All other clusters unaffected.</p>tag:status.pusher.com,2005:Incident/7604722016-09-07T09:37:45Z2016-09-07T09:38:16ZDegraded Dashboard performance<p><small>Sep 7, 09:37 UTC</small><br><strong>Resolved</strong> - Errors have subsided and the dashboard is now serving requests as expected.</p><p><small>Sep 7, 09:02 UTC</small><br><strong>Investigating</strong> - Some customers might experience degraded dashboard performance.</p>tag:status.pusher.com,2005:Incident/5232872016-06-06T09:05:14Z2016-06-06T09:05:15ZAWS Sydney Outage<p><small>Jun 6, 09:05 UTC</small><br><strong>Resolved</strong> - Connectivity in the Asia Pacific region has been stable since being restored yesterday at 1217 UTC.</p><p><small>Jun 5, 12:17 UTC</small><br><strong>Monitoring</strong> - A fix has been implemented and we are monitoring the results.</p><p><small>Jun 5, 06:01 UTC</small><br><strong>Identified</strong> - Customers consuming Pusher services in the Asia Pacific region may be experiencing difficulties connecting to Pusher. This is related to an ongoing multi-availability-zone outage of Amazon AWS in the Sydney area. We are working with our infrastructure provider to restore connectivity as soon as possible and apologise for any inconvenience caused.</p>tag:status.pusher.com,2005:Incident/4635832016-04-18T09:51:32Z2016-04-18T09:51:51ZDatabase upgrade<p><small>Apr 18, 09:51 UTC</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Apr 18, 09:00 UTC</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Apr 11, 16:28 UTC</small><br><strong>Scheduled</strong> - We anticipate that the maintenance window should be less than 1 hour. The status page will be updated as progress is made.
<br />
<br />During this time the Dashboard will be inaccessible but all other services will continue to function as normal.</p>