tag:trust.pixlee.com,2005:/historyPixlee Status - Incident History2018-03-19T07:05:15-07:00Pixleetag:trust.pixlee.com,2005:Incident/16290312018-03-05T13:30:38-08:002018-03-05T13:30:38-08:00Email reports lacking data<p><small>Mar 5, 13:30 PST</small><br><strong>Resolved</strong> - All email reports have been completed correctly and successfully and can be accessed via the Reports section fo the analytics. We will continue to monitor to ensure that no faulty analytics reports remain.</p><p><small>Mar 5, 13:20 PST</small><br><strong>Identified</strong> - On the morning of March 5, 2018, we were made aware of a bug that led to email reports having incomplete or no data. We quickly tracked down the bug and repaired it, and are now in the process of recalculating all email reports that were affected. We believe the affected reports were created between February 2018 and now.</p>tag:trust.pixlee.com,2005:Incident/16091662018-02-19T09:21:31-08:002018-02-19T09:21:31-08:00Analytics Data Sync Issues Feb 7 - Feb 15<p><small>Feb 19, 09:21 PST</small><br><strong>Resolved</strong> - On Feb 13, 2018, it was discovered that our analytics server was not properly responding for data after the date of February 7, 2018. As a result, numbers of Opened Widgets, Interacted, Add to Carts, Conversions etc. were less than they should have been. This issue was resolved on Feb 15, 2018 and further checks put in place to mitigate future cases. We re-ran all analytics reports that occurred in this erroneous time period, but did not re-email those reports (they are always accessible via the Reports section).
<br />
<br />Any analytics viewed between this erroneous time period, if recorded, should not be used. Instead, please visit the same time frames again, and the data should be trustworthy.
<br />
<br />To clarify, no data loss occurred. But during this time period the analytics server was not properly observing all stored data.
<br />
<br />We apologize for delivering this message late.</p>tag:trust.pixlee.com,2005:Incident/15524222018-01-03T23:18:33-08:002018-01-03T23:18:33-08:00Zeroes in analytics<p><small>Jan 3, 23:18 PST</small><br><strong>Resolved</strong> - All analytics are up to date and we are recalcuating and resending analytics reports created in between the December 27 2017 - Jan 3 2018 time period.</p><p><small>Jan 3, 19:00 PST</small><br><strong>Identified</strong> - We noticed this morning that our analytics syncing system fell behind by a few days. Analytics since December 27 will currently show zeroes. We are currently resyncing and getting the data up to date. We will update this ticket when syncing is completed.</p>tag:trust.pixlee.com,2005:Incident/15218982017-11-30T09:06:05-08:002017-11-30T09:06:05-08:00500 analytics responses<p><small>Nov 30, 09:06 PST</small><br><strong>Resolved</strong> - The faulty cluster was rebooted and all operations are back to normal. No data was lost, though it may take up to the next hour to fully process and propagate.</p><p><small>Nov 30, 08:55 PST</small><br><strong>Identified</strong> - We have identified the issue which is related to a cluster reboot and are taking the steps to repair.</p><p><small>Nov 30, 08:34 PST</small><br><strong>Investigating</strong> - We are investigating an issue with our analytics server returning errors.</p>tag:trust.pixlee.com,2005:Incident/15193902017-11-28T10:30:46-08:002017-11-28T10:30:46-08:00Email Displays response<p><small>Nov 28, 10:30 PST</small><br><strong>Resolved</strong> - On Monday Nov 27, 12:15, we had a blip in email displays that caused temporarily delayed response time for email displays approximately one minute. We detected no internal issues and don't believe displays were affected.</p>tag:trust.pixlee.com,2005:Incident/15038192017-11-14T12:34:21-08:002017-11-14T12:34:21-08:00Resolved issue with Website Engagement<p><small>Nov 14, 12:34 PST</small><br><strong>Resolved</strong> - Today at 10AM PST, we discovered an issue with our analytics for tracking website engagement metrics. We observed that from Nov 1 to Nov 14, website engagement metrics were missing certain fields that caused them not to be logged properly. As of 12PM PST Nov 14, we have resolved this issue moving forward and added additional checks to ensure this does not happen again. We apologize for the inconvenience.</p>tag:trust.pixlee.com,2005:Incident/14407182017-10-24T10:42:15-07:002017-10-24T10:42:15-07:00Widget caching<p><small>Oct 24, 10:42 PDT</small><br><strong>Resolved</strong> - This morning during a routine update we noticed that a certain percentage of customers had browser cached the widget code, resulting in incompatible changes which may have caused widgets to not show up. We've modified our deployment code to ensure cache busting via timestamping.</p>tag:trust.pixlee.com,2005:Incident/14254102017-10-18T19:19:25-07:002017-10-18T19:19:25-07:00AWS Oregon connectivity issues<p><small>Oct 18, 19:19 PDT</small><br><strong>Resolved</strong> - Our analytics cluster is fully up and running.</p><p><small>Oct 18, 14:20 PDT</small><br><strong>Monitoring</strong> - AWS is having connectivity issues with their Oregon AZ which affects a segment of our analytics cluster https://status.aws.amazon.com/
<br />
<br />We will continually monitor and report ASAP.</p>tag:trust.pixlee.com,2005:Incident/14251702017-10-18T13:03:33-07:002017-10-18T13:03:33-07:00Email and Analytics Downtime<p><small>Oct 18, 13:03 PDT</small><br><strong>Resolved</strong> - At 12:20 PDT October 18, our alerting reported downtime for our analytics and email servers.
<br />
<br />We were eventually able to trace this to an internally misconfigured security setting which has been reversed and resolved. at 12:58PDT, service was restored across the board.
<br />
<br />Apologies for the inconvenience.</p>tag:trust.pixlee.com,2005:Incident/13988382017-10-05T15:58:32-07:002017-10-05T15:58:32-07:00Brief displays outage<p><small>Oct 5, 15:58 PDT</small><br><strong>Resolved</strong> - From 3:30 to 3:35pm October 5 PST, a subset of our displays experienced an outage. As we were recovering, some displays showed a 404 error page. We have resolved the outage and repaired displays to not display any html content should an outage occur again.</p>tag:trust.pixlee.com,2005:Incident/13844312017-09-27T14:02:05-07:002017-09-27T14:02:05-07:00Mosaic Widgets not displaying content occasionally<p><small>Sep 27, 14:02 PDT</small><br><strong>Resolved</strong> - Have confirmed that the fix resolved the issue.</p><p><small>Sep 27, 13:42 PDT</small><br><strong>Identified</strong> - Around 12:30PM PT, we discovered that mosaic-style displays with the infinite load option toggled would occasionally not display content. We discovered that this was due to an initialization sync issue, and are pushing a solution now.</p>tag:trust.pixlee.com,2005:Incident/13340452017-08-21T08:37:20-07:002017-08-21T08:37:20-07:00Slow Dashboard Response<p><small>Aug 21, 08:37 PDT</small><br><strong>Resolved</strong> - This morning, Pixlee users may have experienced show response times in the dashboard. We identified this issue to an unoptimized database index, which was causing queries to time out on occasion. We noticed and repaired this at approximately 8:30AM, resulting in normal operation of the dashboard again.</p>tag:trust.pixlee.com,2005:Incident/13220112017-08-08T13:44:31-07:002017-08-08T13:44:31-07:00Analytics Server Maintenance<p><small>Aug 8, 13:44 PDT</small><br><strong>Resolved</strong> - The analytics backend has been updated and is back up.</p><p><small>Aug 8, 13:33 PDT</small><br><strong>Identified</strong> - We are temporarily bringing down the analytics backend for maintenance. You will temporarily not be able to access your analytics, though all events are still being written. Thank you for your patience.</p>tag:trust.pixlee.com,2005:Incident/12647202017-06-12T18:01:57-07:002017-06-12T18:01:57-07:00Analytics Write Loss<p><small>Jun 12, 18:01 PDT</small><br><strong>Resolved</strong> - Today from 2:10PM to 5:50PM PST, we experienced an outage with AWS spot instances. As a result, conversion and engagement analytics during this time period were lost. Widgets did not go down during this time. The analytics system has fully recovered and we are putting in further redundancies to prevent this in the future.</p>tag:trust.pixlee.com,2005:Incident/12462822017-05-18T15:48:55-07:002017-05-18T15:48:55-07:00Pixlee Dashboard momentarily down<p><small>May 18, 15:48 PDT</small><br><strong>Resolved</strong> - We experienced a hiccup for approximately 4 minutes in which our access to Dashboard became unresponsive. This has been resolved and the dashboard should be back up.</p>tag:trust.pixlee.com,2005:Incident/12171532017-04-27T12:37:13-07:002017-04-27T12:37:14-07:00Dashboard down<p><small>Apr 27, 12:37 PDT</small><br><strong>Resolved</strong> - A bad push was identified and Dashboard was rolled back and active again.</p><p><small>Apr 27, 12:36 PDT</small><br><strong>Investigating</strong> - Pixlee dashboard is currently down. Investigating</p>tag:trust.pixlee.com,2005:Incident/11975552017-04-12T16:31:56-07:002017-04-12T16:31:56-07:00Some widget displays were temporarily hidden<p><small>Apr 12, 16:31 PDT</small><br><strong>Resolved</strong> - At around 2:34pm, a small percentage of database values were incorrectly marked, causing approximately 60 Pixlee widgets to automatically hide themselves. This mistake was detected and repaired at approximately 3:40pm. No data loss occurred at this time. We apologize for the inconvenience.</p>tag:trust.pixlee.com,2005:Incident/11417692017-02-28T16:37:24-08:002017-02-28T16:37:24-08:00AWS S3 issue<p><small>Feb 28, 16:37 PST</small><br><strong>Resolved</strong> - All systems are back up!</p><p><small>Feb 28, 14:18 PST</small><br><strong>Monitoring</strong> - https://status.aws.amazon.com/
<br />
<br />AWS S3 services are recovering as well as all other AWS services. We are awaiting Heroku to take APIs out of maintenance mode.</p><p><small>Feb 28, 10:16 PST</small><br><strong>Update</strong> - Heroku has turned their APIs on maintenance mode. https://status.heroku.com/
<br />
<br />As a result, Pixlee dashboards are down.</p><p><small>Feb 28, 10:06 PST</small><br><strong>Identified</strong> - the US-East-1 bucket for AWS S3 is experiencing increased error rates. We are monitoring our systems for affect.
<br />
<br />http://status.aws.amazon.com/</p>tag:trust.pixlee.com,2005:Incident/11293762017-02-18T21:49:30-08:002017-02-18T21:49:30-08:00Database Maintenance has begun<p><small>Feb 18, 21:49 PST</small><br><strong>Resolved</strong> - Database updates are complete! All systems restored. Thanks for your patience.</p><p><small>Feb 18, 20:53 PST</small><br><strong>Monitoring</strong> - Access to Pixlee dashboards will be blocked for at most two hours.</p>tag:trust.pixlee.com,2005:Incident/10729662017-01-04T17:46:43-08:002017-01-04T17:46:43-08:00Database recovery<p><small>Jan 4, 17:46 PST</small><br><strong>Resolved</strong> - At 5:26PM Pacific, the Pixlee database began to experience slowness in connection. In the next 5 minutes, the database initiated a reboot and recovery process. At 5:31PM Pacific, the database recovered and all apps self rebooted.</p>tag:trust.pixlee.com,2005:Incident/10120122016-12-06T14:41:22-08:002016-12-06T14:41:22-08:00Action Links not Displaying<p><small>Dec 6, 14:41 PST</small><br><strong>Resolved</strong> - This has been resolved as of 2:25PM Pacific Time.</p><p><small>Dec 6, 14:24 PST</small><br><strong>Identified</strong> - A push was made to our widgets at 1:19PM that hid product and action links in lightboxes. The issue has been detected and a fix is being pushed up.</p>tag:trust.pixlee.com,2005:Incident/9771062016-11-02T15:13:53-07:002016-11-02T15:13:53-07:00Lost conversion events<p><small>Nov 2, 15:13 PDT</small><br><strong>Resolved</strong> - From, 2:04 PM to 2:10 PM PT, conversion events were unable to be recorded to our backend due to a faulty push. A configuration variable was set incorrectly resulting in a disconnect from our attribution server to our events server. This issue has been rectified and extra checks put into place. No widgets were affected.</p>tag:trust.pixlee.com,2005:Incident/9754822016-11-01T11:03:00-07:002016-11-02T12:07:43-07:00DNS Postmortem<p><small>Nov 1, 11:03 PDT</small><br><strong>Resolved</strong> - On the morning of Friday, October 21, there was a large scale DDOS attack performed on Dyn DNS. You can read more details about the attack here: https://www.dynstatus.com/incidents/nlr4yrr162t8
<br />
<br />The result of the attack was that a large chunk of high trafficked websites around the internet (Twitter, Heroku, Github, etc) were momentarily and sporadically unaccessible.
<br />
<br />The issue was resolved by 22:17 UTC on Friday October 21.
<br />
<br />Although we received no customer reports indicating lack of access, there was a possibility of customers not being able to access Pixlee widgets, due to the widespread DNS issues.
<br />
<br />We have confirmed since then that all functionality has been restored.
<br />
<br />No data loss occurred.</p>