tag:status.lob.com,2005:/historyLob Status - Incident History2018-02-12T22:26:32-08:00Lobtag:status.lob.com,2005:Incident/16007432018-02-12T22:26:04-08:002018-02-12T22:26:04-08:00TLS Errors with api.lob.com<p><small>Feb 12, 22:26 PST</small><br><strong>Resolved</strong> - The issue has been fully resolved.</p><p><small>Feb 12, 20:52 PST</small><br><strong>Monitoring</strong> - Earlier today, we had some issues with our TLS configuration. Older languages/HTTP clients were having trouble establishing the TLS handshake with our servers, resulting in errors that wouldn't show up in your Dashboard logs. The issue has been mitigated and is actively being monitored.</p>tag:status.lob.com,2005:Incident/15445742018-01-01T15:07:36-08:002018-01-01T15:07:36-08:00Dashboard Overview, Search, and Logs Misreporting<p><small>Jan 1, 15:07 PST</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Dec 21, 18:47 PST</small><br><strong>Identified</strong> - The Dashboard Overview graph is currently not reporting numbers correctly. All requests are still coming through fine, which you can verify by looking at your Postcard/Letter/Check pages.
<br />
<br />You may also see missing records for both the Search and Logs pages.</p>tag:status.lob.com,2005:Incident/15096532017-11-19T15:59:46-08:002017-11-19T15:59:46-08:00International Verification Downtime<p><small>Nov 19, 15:59 PST</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Nov 19, 12:32 PST</small><br><strong>Identified</strong> - There is an issue affecting International Verifications that has been identified. US Verifications continue to function as expected.</p>tag:status.lob.com,2005:Incident/14254682017-10-21T05:00:40-07:002017-10-21T05:00:40-07:00International Verification Downtime<p><small>Oct 21, 05:00 PDT</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Oct 21, 03:00 PDT</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Oct 18, 14:39 PDT</small><br><strong>Scheduled</strong> - There will be scheduled downtime for our International Verification API from 3am to 5am PDT on Saturday, October 21st. Calls to the International Verification API during this time will err. Domestic (US) verifications will continue to work as expected.</p>tag:status.lob.com,2005:Incident/14253452017-10-18T14:42:59-07:002017-10-18T14:42:59-07:00Connectivity Issues<p><small>Oct 18, 14:42 PDT</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Oct 18, 14:09 PDT</small><br><strong>Identified</strong> - You may experience network connectivity issues with the API due to a slight AWS outage.</p>tag:status.lob.com,2005:Incident/14197092017-10-16T16:58:19-07:002017-10-16T16:58:19-07:00Dashboard Overview Reporting Incorrectly<p><small>Oct 16, 16:58 PDT</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Oct 16, 13:03 PDT</small><br><strong>Identified</strong> - The issue has been identified and a fix is being implemented.</p><p><small>Oct 16, 11:50 PDT</small><br><strong>Investigating</strong> - The Dashboard Overview graph is not reporting numbers correctly for the past few days. All requests are still coming through fine, which you can verify by looking at your Postcard/Letter/Check pages. The issue is being investigated.</p>tag:status.lob.com,2005:Incident/14066752017-10-12T08:48:14-07:002017-10-12T08:48:14-07:00Lob emails not sending<p><small>Oct 12, 08:48 PDT</small><br><strong>Resolved</strong> - The issue has been resolved and emails are sending as expected.</p><p><small>Oct 10, 12:06 PDT</small><br><strong>Identified</strong> - Requested emails such as email verification, password reset, email change, inviting a teammate, and exporting are not triggering properly. The issue has been identified and we are working on resolving it.
<br />
<br />In the meantime, if you need the contents of the email in a timely fashion, please reach out to support@lob.com.</p>tag:status.lob.com,2005:Incident/13516912017-09-13T14:21:21-07:002017-09-13T14:21:21-07:00Dashboard Overview and Search Lag<p><small>Sep 13, 14:21 PDT</small><br><strong>Resolved</strong> - The issue has been resolved and all information should be up to date.</p><p><small>Sep 8, 11:04 PDT</small><br><strong>Identified</strong> - The API Request Volume graph on the Dashboard Overview (https://dashboard.lob.com/#/) and Search functionality is out of date and may return incorrect results.</p>tag:status.lob.com,2005:Incident/13482622017-09-05T16:35:21-07:002017-09-05T16:35:21-07:00Elevated error rates<p><small>Sep 5, 16:35 PDT</small><br><strong>Resolved</strong> - The issue has been resolved.</p><p><small>Sep 5, 16:23 PDT</small><br><strong>Identified</strong> - We've identified a bug that is causing elevated 422 error rates. A fix will be deployed shortly.</p>tag:status.lob.com,2005:Incident/13165302017-08-19T23:30:43-07:002017-08-19T23:30:43-07:00Scheduled Security Maintenence<p><small>Aug 19, 23:30 PDT</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Aug 19, 23:00 PDT</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Aug 3, 14:34 PDT</small><br><strong>Scheduled</strong> - Lob is scheduling a 30 minute window on Saturday, Aug 19th for maintenence to enhance the security of our platform.
<br />
<br />During this window, all Lob API endpoints may return 503 errors. **We do not recommend sending at all during this time.** We suggest either pausing requests for this window, or having some sort of error logging in place so that the requests can be retried at a later time.
<br />
<br />Additionally, the Lob Dashboard may not function as expected and we do not recommend logging in during this window.</p>tag:status.lob.com,2005:Incident/13140602017-08-19T06:00:15-07:002017-08-19T06:00:15-07:00International Verification Downtime<p><small>Aug 19, 06:00 PDT</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Aug 19, 05:00 PDT</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Aug 1, 16:54 PDT</small><br><strong>Scheduled</strong> - There will be scheduled downtime for our International Verification API from 5am to 6am PDT on Saturday, Aug 19th. Calls to the International Verification API during this time will err. Domestic (US) verifications will continue to work as expected.</p>tag:status.lob.com,2005:Incident/12853882017-07-22T05:00:34-07:002017-07-22T05:00:34-07:00International Verification Downtime<p><small>Jul 22, 05:00 PDT</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Jul 22, 04:00 PDT</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Jul 5, 18:33 PDT</small><br><strong>Scheduled</strong> - There will be scheduled downtime for our International Verification API from 4am to 5pm PDT on Saturday, July 22nd. Calls to the International Verification API during this time will err. Domestic (US) verifications will continue to work as expected.</p>tag:status.lob.com,2005:Incident/12592452017-06-17T05:00:28-07:002017-06-17T05:00:28-07:00International Verification Downtime<p><small>Jun 17, 05:00 PDT</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Jun 17, 04:00 PDT</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Jun 5, 16:38 PDT</small><br><strong>Scheduled</strong> - There will be scheduled downtime for our International Verification API from 4am to 5pm PDT on Saturday, June 17th. Calls to the International Verification API during this time will err. Domestic (US) verifications will continue to work as expected.</p>tag:status.lob.com,2005:Incident/12677002017-06-15T17:05:54-07:002017-06-15T17:05:55-07:00Dashboard Volume Graphs & Search missing information<p><small>Jun 15, 17:05 PDT</small><br><strong>Resolved</strong> - Data for Dashboard Volume Graphs and Search are now completely up to date and fully functional.</p><p><small>Jun 15, 14:34 PDT</small><br><strong>Identified</strong> - Volume graphs on the Dashboard Overview page and Dashboard Search are missing some information from the past 24 hours or so. Historical data is currently backfilling. Once all data has been backfilled, we will mark the incident as resolved.</p>tag:status.lob.com,2005:Incident/12278152017-05-20T05:00:38-07:002017-05-20T05:00:38-07:00International Verification Downtime<p><small>May 20, 05:00 PDT</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>May 20, 04:01 PDT</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>May 5, 15:41 PDT</small><br><strong>Scheduled</strong> - There will be scheduled downtime for our International Verification API from 4am to 5pm PDT on Saturday, May 20th. Calls to the International Verification API during this time will err. Domestic (US) verifications will continue to work as expected.</p>tag:status.lob.com,2005:Incident/12452432017-05-17T19:20:24-07:002017-05-17T19:20:24-07:00Lob.com and Dashboard Outage<p><small>May 17, 19:20 PDT</small><br><strong>Resolved</strong> - This issue has been resolved.</p><p><small>May 17, 19:01 PDT</small><br><strong>Investigating</strong> - Lob.com and Dashboard are not functioning - the issue is currently being investigated. The API is fully functional.</p>tag:status.lob.com,2005:Incident/11840522017-04-15T05:00:33-07:002017-04-15T05:00:33-07:00International Verification Downtime<p><small>Apr 15, 05:00 PDT</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Apr 15, 04:00 PDT</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Apr 3, 13:48 PDT</small><br><strong>Scheduled</strong> - There will be scheduled downtime for our International Verification API from 4am to 5pm PDT on Saturday, April 15th. Calls to the International Verification API during this time will err. Domestic (US) verifications will continue to work as expected.</p>tag:status.lob.com,2005:Incident/11403352017-03-11T05:00:24-08:002017-03-11T05:00:24-08:00International Verification Downtime<p><small>Mar 11, 05:00 PST</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Mar 11, 03:00 PST</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Feb 27, 09:48 PST</small><br><strong>Scheduled</strong> - There will be scheduled downtime for our International Verification API (/v1/verify) from 3am to 5am PST on Saturday, March 11th. Calls to the International Verification API during this time will err. Domestic (US) verifications will continue to work as expected.</p>tag:status.lob.com,2005:Incident/11124822017-02-04T12:32:07-08:002017-02-04T12:32:07-08:00Search Outage<p><small>Feb 4, 12:32 PST</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Feb 3, 16:04 PST</small><br><strong>Monitoring</strong> - Search and Dashboard Overview are no longer erring, but historical data is still backfilling. Once all data has been backfilled, we will mark the incident as resolved.</p><p><small>Feb 3, 15:17 PST</small><br><strong>Identified</strong> - Issue has been identified and is in the process of being resolved. Search functionality and Dashboard Overview graph should be available again soon.</p><p><small>Feb 3, 14:06 PST</small><br><strong>Investigating</strong> - We're having issues with our search functionality. Investigation is in progress.</p>tag:status.lob.com,2005:Incident/10053632016-12-01T12:33:16-08:002016-12-01T12:33:16-08:00Increased response times and elevated 504 rate<p><small>Dec 1, 12:33 PST</small><br><strong>Resolved</strong> - The issue has been resolved.</p><p><small>Dec 1, 09:11 PST</small><br><strong>Monitoring</strong> - The issue seems to have been identified and resolved, but we are continuing monitoring.</p><p><small>Nov 30, 23:23 PST</small><br><strong>Investigating</strong> - We are currently investigating this issue.</p>tag:status.lob.com,2005:Incident/7428192016-08-19T11:14:21-07:002016-08-19T11:14:21-07:00/v1/verify Outage<p><small>Aug 19, 11:14 PDT</small><br><strong>Resolved</strong> - The issue has been resolved and /v1/verify is now functioning as normal.</p><p><small>Aug 19, 10:42 PDT</small><br><strong>Investigating</strong> - We are currently investigating an outage with the /v1/verify endpoint. All other endpoints remain functional.</p>tag:status.lob.com,2005:Incident/7396612016-08-16T10:32:41-07:002016-08-16T10:32:41-07:00Elevated Error Rates<p><small>Aug 16, 10:32 PDT</small><br><strong>Resolved</strong> - This was an intermittent issue that has been resolved.</p><p><small>Aug 16, 10:27 PDT</small><br><strong>Investigating</strong> - Our API is responding with an elevated rate of errors. We are investigating the issue.</p>tag:status.lob.com,2005:Incident/5449952016-06-28T19:32:03-07:002016-06-28T19:32:03-07:00Outage: lob.com, dashboard.lob.com<p><small>Jun 28, 19:32 PDT</small><br><strong>Resolved</strong> - The issue has been resolved.</p><p><small>Jun 28, 18:46 PDT</small><br><strong>Identified</strong> - The issue has been identified and we are working on a fix.</p><p><small>Jun 28, 17:52 PDT</small><br><strong>Investigating</strong> - Both lob.com and dashboard.lob.com are currently not responding to requests. We are investigating and will report back when we know more.
<br />
<br />The API is not affected and is fully operational.</p>tag:status.lob.com,2005:Incident/5405102016-06-23T10:24:03-07:002016-06-23T10:24:03-07:00API POST /v1/verify Outage<p><small>Jun 23, 10:24 PDT</small><br><strong>Resolved</strong> - Our API has recovered from an outage affecting the POST /v1/verify endpoint from approximately 12:30 AM to 2:30 AM PST on June 23, 2016.</p>tag:status.lob.com,2005:Incident/5397482016-06-22T15:45:12-07:002016-06-22T15:45:12-07:00Network Downtime<p><small>Jun 22, 15:45 PDT</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Jun 22, 15:28 PDT</small><br><strong>Identified</strong> - Working to resolve network issues affecting the API.</p>