tag:www.papertrailstatus.com,2005:/historyPapertrail Status - Incident History2019-03-21T09:56:51-07:00Papertrailtag:www.papertrailstatus.com,2005:Incident/22691312019-03-19T08:53:28-07:002019-03-19T09:51:14-07:00Log Search and Search Alerts are delayed<p><small>Mar <var data-var='date'>19</var>, <var data-var='time'>08:53</var> PDT</small><br><strong>Resolved</strong> - Logs ingested from 15:05 UTC to 15:36 UTC were not processed by Papertrail's alert system. Therefore, alerts will not run for messages from this time range. This gap only affects alerts, so all messages are available in the viewer, and will also be written to archives.</p><p><small>Mar <var data-var='date'>19</var>, <var data-var='time'>08:13</var> PDT</small><br><strong>Monitoring</strong> - A fix has been implemented and we are monitoring the results.</p><p><small>Mar <var data-var='date'>19</var>, <var data-var='time'>07:59</var> PDT</small><br><strong>Investigating</strong> - We are currently investigating this issue.</p>tag:www.papertrailstatus.com,2005:Incident/22476492019-03-07T17:08:53-08:002019-03-07T17:08:53-08:00The logs6.papertrailapp.com endpoint has increased errors<p><small>Mar <var data-var='date'> 7</var>, <var data-var='time'>17:08</var> PST</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Mar <var data-var='date'> 7</var>, <var data-var='time'>16:50</var> PST</small><br><strong>Update</strong> - We are continuing to monitor for any further issues.</p><p><small>Mar <var data-var='date'> 7</var>, <var data-var='time'>15:51</var> PST</small><br><strong>Monitoring</strong> - The logs6.papertrailapp.com endpoint has returned to normal but we continue to monitor it.</p><p><small>Mar <var data-var='date'> 7</var>, <var data-var='time'>14:46</var> PST</small><br><strong>Investigating</strong> - We are currently investigating this issue.</p>tag:www.papertrailstatus.com,2005:Incident/22467232019-03-07T07:57:02-08:002019-03-07T07:57:02-08:00Connections to the website have an increased error rate<p><small>Mar <var data-var='date'> 7</var>, <var data-var='time'>07:57</var> PST</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Mar <var data-var='date'> 7</var>, <var data-var='time'>07:56</var> PST</small><br><strong>Update</strong> - From 15:00 to 15:10 UTC there were increased error rates on the Papertrail web site. Everything is now back to normal.</p><p><small>Mar <var data-var='date'> 7</var>, <var data-var='time'>07:27</var> PST</small><br><strong>Monitoring</strong> - A fix has been implemented and we are monitoring the results.</p><p><small>Mar <var data-var='date'> 7</var>, <var data-var='time'>07:16</var> PST</small><br><strong>Investigating</strong> - We are currently investigating this issue.</p>tag:www.papertrailstatus.com,2005:Incident/22132872019-02-18T21:03:16-08:002019-02-18T21:03:17-08:00Delays in Log Search and Search Alerts<p><small>Feb <var data-var='date'>18</var>, <var data-var='time'>21:03</var> PST</small><br><strong>Resolved</strong> - Log Searching and Search Alerts have returned to normal performance.</p><p><small>Feb <var data-var='date'>18</var>, <var data-var='time'>19:50</var> PST</small><br><strong>Update</strong> - We are continuing to monitor for any further issues.</p><p><small>Feb <var data-var='date'>18</var>, <var data-var='time'>19:48</var> PST</small><br><strong>Monitoring</strong> - We've identified the problem and performance has been normal for the past hour.</p><p><small>Feb <var data-var='date'>18</var>, <var data-var='time'>18:34</var> PST</small><br><strong>Investigating</strong> - We are currently investigating this issue.</p>tag:www.papertrailstatus.com,2005:Incident/22128272019-02-18T16:54:03-08:002019-02-18T16:54:03-08:00Intermittent short delays in Log Search and Search Alerts<p><small>Feb <var data-var='date'>18</var>, <var data-var='time'>16:54</var> PST</small><br><strong>Resolved</strong> - The behavior of Log Searches and Search Alerts has returned to normal.</p><p><small>Feb <var data-var='date'>18</var>, <var data-var='time'>16:10</var> PST</small><br><strong>Monitoring</strong> - We have deployed a fix and are monitoring the situation. All Log Searches and Search Alerts are back to normal behavior.</p><p><small>Feb <var data-var='date'>18</var>, <var data-var='time'>14:26</var> PST</small><br><strong>Update</strong> - We have identified a cause and we are working on a resolution.</p><p><small>Feb <var data-var='date'>18</var>, <var data-var='time'>12:50</var> PST</small><br><strong>Update</strong> - We continue to investigate.</p><p><small>Feb <var data-var='date'>18</var>, <var data-var='time'>11:51</var> PST</small><br><strong>Update</strong> - We continue to investigate and the delay in Log Search and Search Alerts is impacting most customers now.</p><p><small>Feb <var data-var='date'>18</var>, <var data-var='time'>11:01</var> PST</small><br><strong>Update</strong> - We continue to investigate.</p><p><small>Feb <var data-var='date'>18</var>, <var data-var='time'>09:41</var> PST</small><br><strong>Identified</strong> - Starting at 10:10 UTC on Monday February 18th there have been short intermittent delays for a small number of customers in Log Searching and Search Alerts.</p>tag:www.papertrailstatus.com,2005:Incident/22121412019-02-18T01:52:52-08:002019-02-18T01:52:53-08:00Logs searching and search alerts delayed for a small number of customers<p><small>Feb <var data-var='date'>18</var>, <var data-var='time'>01:52</var> PST</small><br><strong>Resolved</strong> - The backlog has been processed and Log Search and Search Alerts are back to normal behavior.</p><p><small>Feb <var data-var='date'>18</var>, <var data-var='time'>01:42</var> PST</small><br><strong>Monitoring</strong> - We've identified the problem and the backlog is being processed now.</p><p><small>Feb <var data-var='date'>18</var>, <var data-var='time'>00:43</var> PST</small><br><strong>Investigating</strong> - A small number of customers are experiencing delayed log search and search alerts</p>tag:www.papertrailstatus.com,2005:Incident/22102762019-02-16T02:21:13-08:002019-02-16T02:21:14-08:00Search Alerts are delayed<p><small>Feb <var data-var='date'>16</var>, <var data-var='time'>02:21</var> PST</small><br><strong>Resolved</strong> - Search Alerts have been returned to normal behavior.</p><p><small>Feb <var data-var='date'>16</var>, <var data-var='time'>00:12</var> PST</small><br><strong>Monitoring</strong> - Starting at 03:00 UTC on Feb 16th Search Alerts were delayed and we are continuing to work through the backlog.<br /><br />Nearly all of the search alerts that would normally run between 03:00 UTC and 5:45 UTC were not run. A small number of search alerts that would normally run during that time searched a span of days rather than minutes or hours and may have resulted in alerts that were false positives.<br /><br />The backlog of search alerts dating back to 5:45 UTC is being processed now.</p><p><small>Feb <var data-var='date'>15</var>, <var data-var='time'>23:14</var> PST</small><br><strong>Identified</strong> - We've identified the problem and we're working to get Search Alerts back to realtime.</p><p><small>Feb <var data-var='date'>15</var>, <var data-var='time'>21:50</var> PST</small><br><strong>Update</strong> - We continue to investigate.</p><p><small>Feb <var data-var='date'>15</var>, <var data-var='time'>20:36</var> PST</small><br><strong>Update</strong> - We are continuing to investigate and search alerts remain delayed.</p><p><small>Feb <var data-var='date'>15</var>, <var data-var='time'>19:21</var> PST</small><br><strong>Investigating</strong> - Search alerts are delayed.</p>tag:www.papertrailstatus.com,2005:Incident/21758122019-02-14T14:14:36-08:002019-02-14T14:14:37-08:00Slower TLS handshakes on logs.papertrailapp.com and logs6.papertrailapp.com<p><small>Feb <var data-var='date'>14</var>, <var data-var='time'>14:14</var> PST</small><br><strong>Resolved</strong> - Additional resources were added to the logs and logs6 endpoints and they have been stable for a week. We will continue to monitor, but do not expect any further customer impact from slow TLS handshakes.</p><p><small>Jan <var data-var='date'>28</var>, <var data-var='time'>15:57</var> PST</small><br><strong>Monitoring</strong> - The logs and logs6 endpoints have been experiencing increased load recently, leading to slower TLS handshakes during peak periods.<br /><br />UDP senders and plaintext TCP senders are not affected. Some TLS senders may see increased queueing and delays during peak times, depending on how often they reconnect to Papertrail. Some senders could even drop messages, in the absence of client-side queueing.<br /><br />We're adding capacity to handle the increased load, but it may be a few days before that is online. In the meantime, anyone affected by the slower TLS handshakes is advised to create a new destination, which will be allocated on a different cluster.</p>tag:www.papertrailstatus.com,2005:Incident/22041342019-02-12T15:33:10-08:002019-02-12T15:33:10-08:00Search alerts are delayed<p><small>Feb <var data-var='date'>12</var>, <var data-var='time'>15:33</var> PST</small><br><strong>Resolved</strong> - From 19:17 UTC to 20:05 UTC search alerts were delayed.</p><p><small>Feb <var data-var='date'>12</var>, <var data-var='time'>13:06</var> PST</small><br><strong>Monitoring</strong> - Search alerts have returned to normal.</p><p><small>Feb <var data-var='date'>12</var>, <var data-var='time'>12:03</var> PST</small><br><strong>Investigating</strong> - Search alerts have been delayed since 19:17 UTC.</p>tag:www.papertrailstatus.com,2005:Incident/22008752019-02-10T23:37:37-08:002019-02-10T23:44:09-08:00Partial log processing delay & usage statistics innacuracies<p><small>Feb <var data-var='date'>10</var>, <var data-var='time'>23:37</var> PST</small><br><strong>Resolved</strong> - Accrued usage (in bytes) and "last event seen" counters are working as of 07:25 UTC on 11 Feb 2019. Bytes received after that time and sources seen after that time will be tracked as they were before this incident.</p><p><small>Feb <var data-var='date'>10</var>, <var data-var='time'>22:07</var> PST</small><br><strong>Identified</strong> - We've identified the blockage and have returned logs to real-time status. Some 24-hour search alerts may not reflect accurate message counts.<br /><br />Accrued usage (in bytes) and "last event seen" counters may not reflect accurate information for some customers. Engineering is actively working on restoring this functionality. This does not impact log processing/ingestion, search, or alerts.</p><p><small>Feb <var data-var='date'>10</var>, <var data-var='time'>20:57</var> PST</small><br><strong>Update</strong> - Engineering is still investigating. We've determined the scope to be contained to larger-volume customers. The effect is a cease of new logs in the Events viewer as well as Search Alerts not triggering. Log collection is NOT impacted so when the delay catches up, logs received during this period will become available.</p><p><small>Feb <var data-var='date'>10</var>, <var data-var='time'>19:48</var> PST</small><br><strong>Update</strong> - A subset of customers will see delay in events viewer. We are still investigating the cause of the log processing delay.</p><p><small>Feb <var data-var='date'>10</var>, <var data-var='time'>19:01</var> PST</small><br><strong>Update</strong> - We are still investigating the cause of the log processing delay and trying to process the backlog.</p><p><small>Feb <var data-var='date'>10</var>, <var data-var='time'>17:38</var> PST</small><br><strong>Investigating</strong> - We are currently investigating log processing delay.</p>tag:www.papertrailstatus.com,2005:Incident/21857412019-02-01T14:31:47-08:002019-02-01T14:31:47-08:00logs6.papertrailapp.com errors<p><small>Feb <var data-var='date'> 1</var>, <var data-var='time'>14:31</var> PST</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Feb <var data-var='date'> 1</var>, <var data-var='time'>12:55</var> PST</small><br><strong>Monitoring</strong> - The logs6.papertrailapp.com endpoint has been returned to normal operation.</p><p><small>Feb <var data-var='date'> 1</var>, <var data-var='time'>12:43</var> PST</small><br><strong>Identified</strong> - Connectivity to logs6.papertrailapp.com is being restored.</p><p><small>Feb <var data-var='date'> 1</var>, <var data-var='time'>12:36</var> PST</small><br><strong>Investigating</strong> - We are currently investigating this issue.</p>tag:www.papertrailstatus.com,2005:Incident/21832032019-01-31T16:19:00-08:002019-01-31T16:20:39-08:00Delays in log processing pipeline<p><small>Jan <var data-var='date'>31</var>, <var data-var='time'>16:19</var> PST</small><br><strong>Resolved</strong> - The ingestion pipeline has been healthy for 3 hours, and we expect no further delays today. Should things change, we'll notify everyone appropriately.</p><p><small>Jan <var data-var='date'>31</var>, <var data-var='time'>12:20</var> PST</small><br><strong>Update</strong> - We continue to have periodic delays in the pipeline that quickly resolve. We're keeping this incident open while we continue investigating.</p><p><small>Jan <var data-var='date'>31</var>, <var data-var='time'>11:16</var> PST</small><br><strong>Investigating</strong> - We are currently investigating an intermittent log processing delay.</p>tag:www.papertrailstatus.com,2005:Incident/21831072019-01-31T10:55:54-08:002019-01-31T10:55:55-08:00Delays in log processing pipeline<p><small>Jan <var data-var='date'>31</var>, <var data-var='time'>10:55</var> PST</small><br><strong>Resolved</strong> - The backlog has been processed and we're back to realtime.</p><p><small>Jan <var data-var='date'>31</var>, <var data-var='time'>10:40</var> PST</small><br><strong>Investigating</strong> - We are currently investigating this issue.</p>tag:www.papertrailstatus.com,2005:Incident/21826942019-01-31T07:20:55-08:002019-01-31T08:11:56-08:00Delays in log processing pipeline<p><small>Jan <var data-var='date'>31</var>, <var data-var='time'>07:20</var> PST</small><br><strong>Resolved</strong> - From 15:22 to 16:03 UTC the log processing pipeline was delayed. Search Alerts and Log Viewing were impacted by this delay but the backlog has been processed and logs are back to realtime.</p>tag:www.papertrailstatus.com,2005:Incident/21775792019-01-29T11:04:39-08:002019-01-29T13:49:57-08:00papertrailapp.com errors, delayed log collection, searching and alerts<p><small>Jan <var data-var='date'>29</var>, <var data-var='time'>11:04</var> PST</small><br><strong>Resolved</strong> - From 17:45 UTC to 18:25 UTC requests to papertrailapp.com were delayed or timed out. Log Collection, Log Searching and Search Alerts were periodically delayed. The backlog of logs, searches and alerts were processed by 18:45 UTC.</p><p><small>Jan <var data-var='date'>29</var>, <var data-var='time'>10:57</var> PST</small><br><strong>Monitoring</strong> - The backlog of alerts have been processed and we're continuing to monitor the situation.</p><p><small>Jan <var data-var='date'>29</var>, <var data-var='time'>09:20</var> PST</small><br><strong>Investigating</strong> - We are currently investigating this issue.</p>tag:www.papertrailstatus.com,2005:Incident/21772342019-01-29T08:40:15-08:002019-01-29T08:40:15-08:00Log processing delay<p><small>Jan <var data-var='date'>29</var>, <var data-var='time'>08:40</var> PST</small><br><strong>Resolved</strong> - Backlog has been processed. We are back to normal. If you experience any issues, please reach out to us at support@papertrailapp.com.</p><p><small>Jan <var data-var='date'>29</var>, <var data-var='time'>08:07</var> PST</small><br><strong>Monitoring</strong> - We are starting to process the backlog</p><p><small>Jan <var data-var='date'>29</var>, <var data-var='time'>07:31</var> PST</small><br><strong>Investigating</strong> - We are currently investigating log processing delay.</p>tag:www.papertrailstatus.com,2005:Incident/21689112019-01-25T03:35:53-08:002019-01-25T03:35:59-08:00Web errors and delays processing logs<p><small>Jan <var data-var='date'>25</var>, <var data-var='time'>03:35</var> PST</small><br><strong>Resolved</strong> - Search alerts and log search are back to realtime and website errors are resolved.</p><p><small>Jan <var data-var='date'>25</var>, <var data-var='time'>02:53</var> PST</small><br><strong>Update</strong> - We continue to work on the issue. Log collection is not impacted.</p><p><small>Jan <var data-var='date'>25</var>, <var data-var='time'>01:35</var> PST</small><br><strong>Investigating</strong> - We are currently investigating this issue.</p>tag:www.papertrailstatus.com,2005:Incident/21610722019-01-21T16:37:07-08:002019-01-21T16:37:07-08:00Event Viewer incident<p><small>Jan <var data-var='date'>21</var>, <var data-var='time'>16:37</var> PST</small><br><strong>Resolved</strong> - From 23:15 - 00:32 UTC we had a disruption in the log viewer and search alerts. This issue has been fixed. If you experience any issues, please reach out to us at support@papertrailapp.com.</p><p><small>Jan <var data-var='date'>21</var>, <var data-var='time'>16:22</var> PST</small><br><strong>Investigating</strong> - We are currently investigating an issue with event viewer.</p>tag:www.papertrailstatus.com,2005:Incident/21534502019-01-17T01:21:02-08:002019-01-17T01:21:02-08:00Search alert processing delay<p><small>Jan <var data-var='date'>17</var>, <var data-var='time'>01:21</var> PST</small><br><strong>Resolved</strong> - We've finished processing Delay search alert backlog. This incident has been resolved.</p><p><small>Jan <var data-var='date'>17</var>, <var data-var='time'>00:43</var> PST</small><br><strong>Update</strong> - All alerts are back to realtime, but we are still processing a backlog of old alerts.</p><p><small>Jan <var data-var='date'>16</var>, <var data-var='time'>22:24</var> PST</small><br><strong>Monitoring</strong> - The Minute alerts are back to real time. We are still burning down the backlog for Delay search alerts.</p><p><small>Jan <var data-var='date'>16</var>, <var data-var='time'>20:31</var> PST</small><br><strong>Investigating</strong> - Search alerts for all customers are delayed. We're working on it.</p>tag:www.papertrailstatus.com,2005:Incident/21448972019-01-12T18:16:27-08:002019-01-12T18:16:27-08:00Partial log processing delay<p><small>Jan <var data-var='date'>12</var>, <var data-var='time'>18:16</var> PST</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Jan <var data-var='date'>12</var>, <var data-var='time'>16:31</var> PST</small><br><strong>Monitoring</strong> - Backlog has been processed. We're keeping this open but at the time everything should be normal.</p><p><small>Jan <var data-var='date'>12</var>, <var data-var='time'>16:13</var> PST</small><br><strong>Investigating</strong> - We are currently investigating log processing delay. A subset of customers will be effected.</p>tag:www.papertrailstatus.com,2005:Incident/21441122019-01-12T01:30:42-08:002019-01-12T01:30:43-08:00Partial log processing delay<p><small>Jan <var data-var='date'>12</var>, <var data-var='time'>01:30</var> PST</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Jan <var data-var='date'>12</var>, <var data-var='time'>00:12</var> PST</small><br><strong>Monitoring</strong> - Backlog has been processed. We're keeping this open but at the time everything should be normal.</p><p><small>Jan <var data-var='date'>12</var>, <var data-var='time'>00:02</var> PST</small><br><strong>Investigating</strong> - We are currently investigating log processing delay. A subset of customers will be effected.</p>tag:www.papertrailstatus.com,2005:Incident/21431502019-01-11T09:54:15-08:002019-01-11T12:25:56-08:00Partial log processing delay<p><small>Jan <var data-var='date'>11</var>, <var data-var='time'>09:54</var> PST</small><br><strong>Resolved</strong> - Some customers experienced delays in the log viewer starting at 9:54 AM PST. Everything caught up within 30 minutes, by 10:24 AM PST.</p>tag:www.papertrailstatus.com,2005:Incident/21399892019-01-10T08:38:52-08:002019-01-10T08:38:53-08:00Hosting Provider networking issues<p><small>Jan <var data-var='date'>10</var>, <var data-var='time'>08:38</var> PST</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Jan <var data-var='date'>10</var>, <var data-var='time'>07:32</var> PST</small><br><strong>Monitoring</strong> - The network has been stable for the past hour. We're keeping this open but at the time everything should be normal.</p><p><small>Jan <var data-var='date'>10</var>, <var data-var='time'>06:34</var> PST</small><br><strong>Identified</strong> - Our hosting provider is having network issues. You might see lag in search alerts and log viewer. We're investigating.</p>tag:www.papertrailstatus.com,2005:Incident/21346362019-01-07T17:45:39-08:002019-01-07T17:45:40-08:00Partial log processing delay<p><small>Jan <var data-var='date'> 7</var>, <var data-var='time'>17:45</var> PST</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Jan <var data-var='date'> 7</var>, <var data-var='time'>17:45</var> PST</small><br><strong>Monitoring</strong> - A fix has been implemented and we are monitoring the results.</p><p><small>Jan <var data-var='date'> 7</var>, <var data-var='time'>17:42</var> PST</small><br><strong>Identified</strong> - The issue has been identified and a fix is being implemented.</p><p><small>Jan <var data-var='date'> 7</var>, <var data-var='time'>17:19</var> PST</small><br><strong>Investigating</strong> - We are currently investigating log processing delay. A subset of customers will be effected.</p>tag:www.papertrailstatus.com,2005:Incident/21335212019-01-07T13:26:04-08:002019-01-07T13:26:05-08:00Partial log processing delay<p><small>Jan <var data-var='date'> 7</var>, <var data-var='time'>13:26</var> PST</small><br><strong>Resolved</strong> - Backlogs and search alerts are cleared. We are back to normal! If you experience any issues, please reach out to us at support@papertrailapp.com.</p><p><small>Jan <var data-var='date'> 7</var>, <var data-var='time'>11:15</var> PST</small><br><strong>Update</strong> - We are continuing to work on a fix for this issue.</p><p><small>Jan <var data-var='date'> 7</var>, <var data-var='time'>08:30</var> PST</small><br><strong>Update</strong> - Additional capacity is online and we're starting to burn down the backlog.</p><p><small>Jan <var data-var='date'> 7</var>, <var data-var='time'>07:11</var> PST</small><br><strong>Identified</strong> - We're deploying extra capacity to handle the load spike.</p><p><small>Jan <var data-var='date'> 7</var>, <var data-var='time'>06:52</var> PST</small><br><strong>Investigating</strong> - We are currently investigating log processing delay. A subset of customers will be effected.</p>