tag:status.fairhair.ai,2005:/historyFairhair.ai Status - Incident History2019-09-15T08:07:13ZFairhair.aitag:status.fairhair.ai,2005:Incident/29626122019-09-05T15:01:46Z2019-09-05T15:01:46ZLow data volumes from Reddit<p><small>Sep <var data-var='date'> 5</var>, <var data-var='time'>15:01</var> UTC</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Sep <var data-var='date'> 5</var>, <var data-var='time'>15:01</var> UTC</small><br><strong>Update</strong> - All data that was missed during the low volume period has now been delivered. The Reddit data feed is now operating normally.</p><p><small>Sep <var data-var='date'> 5</var>, <var data-var='time'>13:39</var> UTC</small><br><strong>Monitoring</strong> - Data volumes for reddit have returned to normal. We expect all missed data during the incident period to be delivered.</p><p><small>Sep <var data-var='date'> 5</var>, <var data-var='time'>11:42</var> UTC</small><br><strong>Investigating</strong> - We are receiving a low volume of data on our Reddit feed. We are investigating the issue.</p>tag:status.fairhair.ai,2005:Incident/28506652019-09-05T14:52:59Z2019-09-05T14:53:00ZErroneous reduction in filter usage reporting<p><small>Sep <var data-var='date'> 5</var>, <var data-var='time'>14:52</var> UTC</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Aug <var data-var='date'>21</var>, <var data-var='time'>10:42</var> UTC</small><br><strong>Monitoring</strong> - A fix has been implemented and we are monitoring the results.</p><p><small>Aug <var data-var='date'>12</var>, <var data-var='time'>10:53</var> UTC</small><br><strong>Identified</strong> - We have identified an internal processing issue causing a reduction in the reported DPU amount for filter processing. This issue started 2019-08-07. We are actively working to resolve this issue.</p>tag:status.fairhair.ai,2005:Incident/29244652019-09-02T07:05:45Z2019-09-02T07:05:45ZReceiving low volume of Video, YouTube and Dailymotion data<p><small>Sep <var data-var='date'> 2</var>, <var data-var='time'>07:05</var> UTC</small><br><strong>Investigating</strong> - Video, YouTube and Dailymotion data throughput volumes dropped at 21:00 UTC 1 Sept. We are working with the providers to resolve this issue. It is expected that full data fidelity will be recovered. We will provide updates until the incident is resolved.</p><p><small>Aug <var data-var='date'>30</var>, <var data-var='time'>07:05</var> UTC</small><br><strong>Monitoring</strong> - A fix has been implemented and we are monitoring the results.</p><p><small>Aug <var data-var='date'>28</var>, <var data-var='time'>08:55</var> UTC</small><br><strong>Update</strong> - Video and Dailymotion data is now flowing as expected, we are still seeing low throughput of YouTube data, we will continue working with our provider to resolve this issue.</p><p><small>Aug <var data-var='date'>27</var>, <var data-var='time'>20:53</var> UTC</small><br><strong>Investigating</strong> - Video, YouTube and Dailymotion data throughput volumes dropped at 20:00 UTC. We are working with the providers to resolve this issue. It is expected that full data fidelity will be recovered. We will provide updates until the incident is resolved.</p>tag:status.fairhair.ai,2005:Incident/28861362019-08-19T19:55:18Z2019-08-19T19:55:19ZReceiving low volume of Reddit data<p><small>Aug <var data-var='date'>19</var>, <var data-var='time'>19:55</var> UTC</small><br><strong>Resolved</strong> - This issue was resolved as of 19:10 UTC.</p><p><small>Aug <var data-var='date'>19</var>, <var data-var='time'>17:36</var> UTC</small><br><strong>Investigating</strong> - Reddit data throughput volumes dropped at 1700 UTC. We are working to resolve this issue. It is expected that full data fidelity will be recovered. We will provides updates until the incident is resolved.</p>tag:status.fairhair.ai,2005:Incident/28758112019-08-19T10:27:52Z2019-08-19T10:27:53ZReceiving low volume of LexisNexis data<p><small>Aug <var data-var='date'>19</var>, <var data-var='time'>10:27</var> UTC</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Aug <var data-var='date'>19</var>, <var data-var='time'>07:03</var> UTC</small><br><strong>Monitoring</strong> - A fix has been implemented and we are monitoring the results.</p><p><small>Aug <var data-var='date'>18</var>, <var data-var='time'>12:52</var> UTC</small><br><strong>Update</strong> - We are still working with LexisNexis on this issue, we will provide further updates as we recieve them.</p><p><small>Aug <var data-var='date'>17</var>, <var data-var='time'>08:26</var> UTC</small><br><strong>Investigating</strong> - LexisNexis data throughput volumes dropped at 03:48 UTC. We are working with LexisNexis to resolve this issue. It is expected that full data fidelity will be recovered. We will provide updates until the incident is resolved.</p>tag:status.fairhair.ai,2005:Incident/28550412019-08-13T15:21:53Z2019-08-13T15:21:54ZDelays on delivery to Push Destinations<p><small>Aug <var data-var='date'>13</var>, <var data-var='time'>15:21</var> UTC</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Aug <var data-var='date'>13</var>, <var data-var='time'>13:17</var> UTC</small><br><strong>Identified</strong> - The issue has been identified and a fix is being implemented.</p><p><small>Aug <var data-var='date'>13</var>, <var data-var='time'>06:35</var> UTC</small><br><strong>Investigating</strong> - We are observing increased latency on deliveries to Push Destinations. We are working to resolve this.</p>tag:status.fairhair.ai,2005:Incident/28525202019-08-12T21:00:24Z2019-08-12T21:00:24ZDelays on delivery to Push Destinations<p><small>Aug <var data-var='date'>12</var>, <var data-var='time'>21:00</var> UTC</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Aug <var data-var='date'>12</var>, <var data-var='time'>18:27</var> UTC</small><br><strong>Investigating</strong> - We are observing increased latency on deliveries to Push Destinations. We are working to resolve this.</p>tag:status.fairhair.ai,2005:Incident/28271842019-08-07T12:36:44Z2019-08-07T12:36:45ZErrors starting new filters<p><small>Aug <var data-var='date'> 7</var>, <var data-var='time'>12:36</var> UTC</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Aug <var data-var='date'> 7</var>, <var data-var='time'>10:52</var> UTC</small><br><strong>Monitoring</strong> - A fix has been implemented and we are monitoring the results.</p><p><small>Aug <var data-var='date'> 7</var>, <var data-var='time'>10:31</var> UTC</small><br><strong>Investigating</strong> - We are observing elevated error rates on filter compilation</p>tag:status.fairhair.ai,2005:Incident/28185782019-08-07T01:42:31Z2019-08-07T01:42:31ZNo data from reddit<p><small>Aug <var data-var='date'> 7</var>, <var data-var='time'>01:42</var> UTC</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Aug <var data-var='date'> 5</var>, <var data-var='time'>18:25</var> UTC</small><br><strong>Investigating</strong> - We are experiencing a gap in our data feed from Reddit. Customers will not be receiving any Reddit data in their subscriptions. We are working to resolve the issue. No other feeds are affected.</p>tag:status.fairhair.ai,2005:Incident/28034142019-08-03T07:29:38Z2019-08-03T07:29:39ZDelays on delivery to Push Destinations<p><small>Aug <var data-var='date'> 3</var>, <var data-var='time'>07:29</var> UTC</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Aug <var data-var='date'> 2</var>, <var data-var='time'>14:18</var> UTC</small><br><strong>Monitoring</strong> - A fix has been implemented and we are monitoring the results.</p><p><small>Aug <var data-var='date'> 2</var>, <var data-var='time'>13:46</var> UTC</small><br><strong>Identified</strong> - The issue has been identified and a fix is being implemented.</p><p><small>Aug <var data-var='date'> 2</var>, <var data-var='time'>12:07</var> UTC</small><br><strong>Investigating</strong> - We are observing increased latency on deliveries to Push Destinations. We are working to resolve this.</p>tag:status.fairhair.ai,2005:Incident/27977362019-08-01T20:51:22Z2019-08-01T20:51:23ZDelay in data delivery<p><small>Aug <var data-var='date'> 1</var>, <var data-var='time'>20:51</var> UTC</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Aug <var data-var='date'> 1</var>, <var data-var='time'>14:49</var> UTC</small><br><strong>Monitoring</strong> - A fix has been implemented and we are monitoring the results.</p><p><small>Aug <var data-var='date'> 1</var>, <var data-var='time'>13:58</var> UTC</small><br><strong>Investigating</strong> - We are currently seeing a delay in data delivery, we are investigating and will update accordingly.</p>tag:status.fairhair.ai,2005:Incident/27913712019-07-31T20:39:29Z2019-07-31T20:39:30ZDelays starting or stopping new tasks<p><small>Jul <var data-var='date'>31</var>, <var data-var='time'>20:39</var> UTC</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Jul <var data-var='date'>31</var>, <var data-var='time'>13:50</var> UTC</small><br><strong>Monitoring</strong> - A fix has been implemented and we are monitoring the results.</p><p><small>Jul <var data-var='date'>31</var>, <var data-var='time'>12:15</var> UTC</small><br><strong>Investigating</strong> - We are currently experiencing delays starting and stopping new tasks. We are investigating the issue.</p>tag:status.fairhair.ai,2005:Incident/27811942019-07-29T20:09:23Z2019-07-29T20:09:24ZDelays on delivery to Push Destinations<p><small>Jul <var data-var='date'>29</var>, <var data-var='time'>20:09</var> UTC</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Jul <var data-var='date'>29</var>, <var data-var='time'>16:29</var> UTC</small><br><strong>Monitoring</strong> - We have tracked down the issue to an internal buffering issue. We are now processing a backlog of data.</p><p><small>Jul <var data-var='date'>29</var>, <var data-var='time'>15:37</var> UTC</small><br><strong>Update</strong> - We are continuing to investigate this issue.</p><p><small>Jul <var data-var='date'>29</var>, <var data-var='time'>15:37</var> UTC</small><br><strong>Investigating</strong> - We are observing increased latency on deliveries to Push Destinations. We are working to resolve this.</p>tag:status.fairhair.ai,2005:Incident/27448582019-07-22T12:31:06Z2019-07-22T12:31:07ZDuplicate data and delayed data for social feeds<p><small>Jul <var data-var='date'>22</var>, <var data-var='time'>12:31</var> UTC</small><br><strong>Resolved</strong> - All social feeds are now clear of lag and processing in real time.</p><p><small>Jul <var data-var='date'>22</var>, <var data-var='time'>11:27</var> UTC</small><br><strong>Update</strong> - The following feeds are still seeing delayed data: Disqus, Wordpess, Tumblr<br />All other social feeds are now processing real-time data.</p><p><small>Jul <var data-var='date'>22</var>, <var data-var='time'>10:50</var> UTC</small><br><strong>Identified</strong> - A number of social feeds have seen a quantity of duplicate interactions, resulting in delays delivering data to customers.<br />Impacted data sources include: reddit, tumblr, wordpress, lexisnexis, disqus and intensdebate. We are working to minimise impact to customers.</p>tag:status.fairhair.ai,2005:Incident/27059912019-07-14T23:03:44Z2019-07-14T23:03:45ZReceiving low volume of Reddit, Youtube, Boards, Blogs, Dailymotion, 2ch data<p><small>Jul <var data-var='date'>14</var>, <var data-var='time'>23:03</var> UTC</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Jul <var data-var='date'>14</var>, <var data-var='time'>13:24</var> UTC</small><br><strong>Investigating</strong> - Reddit, Youtube, Boards, Blogs, Dailymotion, 2ch data throughput volumes dropped at 11:40 UTC. We are working with our upstream provider to resolve this issue. It is expected that full data fidelity will be recovered. We will provides updates until the incident is resolved.</p>tag:status.fairhair.ai,2005:Incident/26360382019-07-01T16:13:53Z2019-07-01T16:13:54ZAPI instability<p><small>Jul <var data-var='date'> 1</var>, <var data-var='time'>16:13</var> UTC</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Jul <var data-var='date'> 1</var>, <var data-var='time'>13:42</var> UTC</small><br><strong>Monitoring</strong> - A fix has been implemented and we are monitoring the results.</p><p><small>Jul <var data-var='date'> 1</var>, <var data-var='time'>13:00</var> UTC</small><br><strong>Investigating</strong> - Users may experience errors while attempting to access certain API endpoints. We are investigating the root cause of this.</p>tag:status.fairhair.ai,2005:Incident/26159602019-06-28T11:30:40Z2019-06-28T11:30:40ZTumblr feed maintenance<p><small>Jun <var data-var='date'>28</var>, <var data-var='time'>11:30</var> UTC</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Jun <var data-var='date'>27</var>, <var data-var='time'>15:00</var> UTC</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Jun <var data-var='date'>27</var>, <var data-var='time'>11:20</var> UTC</small><br><strong>Scheduled</strong> - Tumblr data will be unavailable during this window due to upstream maintenance. A backfill of data will be provided once service resumes.</p>tag:status.fairhair.ai,2005:Incident/25443132019-06-17T11:58:53Z2019-06-17T11:58:53ZDelays on delivery to Push Destinations<p><small>Jun <var data-var='date'>17</var>, <var data-var='time'>11:58</var> UTC</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Jun <var data-var='date'>13</var>, <var data-var='time'>19:19</var> UTC</small><br><strong>Investigating</strong> - We are observing increased latency on deliveries to Push Destinations. We are working to resolve this.</p>tag:status.fairhair.ai,2005:Incident/23744382019-05-10T06:52:08Z2019-05-10T06:52:08ZReceiving low volume of Reddit data<p><small>May <var data-var='date'>10</var>, <var data-var='time'>06:52</var> UTC</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>May <var data-var='date'> 9</var>, <var data-var='time'>10:48</var> UTC</small><br><strong>Monitoring</strong> - A fix has been implemented and we are monitoring the results.</p><p><small>May <var data-var='date'> 8</var>, <var data-var='time'>07:29</var> UTC</small><br><strong>Update</strong> - We are continuing to investigate this outage. We are currently waiting for our upstream provider to resolve an issue, we will update further as soon as we have more information.</p><p><small>May <var data-var='date'> 7</var>, <var data-var='time'>13:56</var> UTC</small><br><strong>Update</strong> - We are still investigating this issue and will update when we have further information.</p><p><small>May <var data-var='date'> 7</var>, <var data-var='time'>07:17</var> UTC</small><br><strong>Investigating</strong> - Reddit data throughput volumes dropped at 23:36 UTC. We are working with Reddit to resolve this issue. It is expected that full data fidelity will be recovered. We will provides updates until the incident is resolved.</p>tag:status.fairhair.ai,2005:Incident/23836642019-05-09T19:03:16Z2019-05-09T19:03:17ZStreaming API unavailable<p><small>May <var data-var='date'> 9</var>, <var data-var='time'>19:03</var> UTC</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>May <var data-var='date'> 9</var>, <var data-var='time'>17:53</var> UTC</small><br><strong>Investigating</strong> - We are currently investigating this issue.</p>tag:status.fairhair.ai,2005:Incident/23581972019-04-30T09:05:29Z2019-04-30T09:05:30ZLow data volumes from Disqus<p><small>Apr <var data-var='date'>30</var>, <var data-var='time'>09:05</var> UTC</small><br><strong>Resolved</strong> - The issues affecting disqus feed have been resolved. No data was lost during this issue.</p><p><small>Apr <var data-var='date'>29</var>, <var data-var='time'>21:42</var> UTC</small><br><strong>Investigating</strong> - We are currently experiencing lower than expected traffic on the disqus feed. We are investigating the issue. No other feeds are affected.</p>tag:status.fairhair.ai,2005:Incident/23370152019-04-19T09:30:32Z2019-04-19T09:30:32ZDelays on delivery to Push Destinations<p><small>Apr <var data-var='date'>19</var>, <var data-var='time'>09:30</var> UTC</small><br><strong>Resolved</strong> - Total window of disruption 15:30 - 20:30, however certain feeds were less than this.</p><p><small>Apr <var data-var='date'>18</var>, <var data-var='time'>17:10</var> UTC</small><br><strong>Investigating</strong> - We are observing increased latency on deliveries to Push Destinations for certain sources. We are working to resolve this.</p>tag:status.fairhair.ai,2005:Incident/22606522019-04-12T12:51:02Z2019-04-12T12:51:02ZWordPress data missing fields<p><small>Apr <var data-var='date'>12</var>, <var data-var='time'>12:51</var> UTC</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Mar <var data-var='date'>14</var>, <var data-var='time'>16:59</var> UTC</small><br><strong>Monitoring</strong> - Between 15:05 and 16:45 delivered WordPress data was missing key fields, including wordpress.post_id. We have now resolved this issue and will redeliver the data with all fields populated.</p>tag:status.fairhair.ai,2005:Incident/23220212019-04-11T13:21:03Z2019-04-11T13:21:04ZReceiving low volume of Reddit data<p><small>Apr <var data-var='date'>11</var>, <var data-var='time'>13:21</var> UTC</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Apr <var data-var='date'>11</var>, <var data-var='time'>12:56</var> UTC</small><br><strong>Monitoring</strong> - A fix has been implemented and we are monitoring the results.</p><p><small>Apr <var data-var='date'>11</var>, <var data-var='time'>11:43</var> UTC</small><br><strong>Investigating</strong> - Reddit data throughput volumes dropped at 11:02 UTC. We are working with Reddit to resolve this issue. It is expected that full data fidelity will be recovered. We will provide updates until the incident is resolved.</p>tag:status.fairhair.ai,2005:Incident/23151422019-04-09T13:32:43Z2019-04-09T13:32:44ZReceiving low volume of some social data feeds<p><small>Apr <var data-var='date'> 9</var>, <var data-var='time'>13:32</var> UTC</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Apr <var data-var='date'> 9</var>, <var data-var='time'>10:01</var> UTC</small><br><strong>Monitoring</strong> - A fix has been implemented and we are monitoring the results.</p><p><small>Apr <var data-var='date'> 8</var>, <var data-var='time'>13:14</var> UTC</small><br><strong>Investigating</strong> - We are experiencing low data throughput volumes on the following sources, YouTube, Dailymotion, Reddit, blogs and boards.<br /><br />Data throughput volumes dropped at 12:40 UTC. We are working with our provider to resolve this issue. It is expected that full data fidelity will be recovered. We will provide updates until the incident is resolved.</p>