tag:status.letsfreckle.com,2005:/historyFreckle Time Tracking Status - Incident History2018-02-17T03:06:30-05:00Freckle Time Trackingtag:status.letsfreckle.com,2005:Incident/16051622018-02-15T15:23:05-05:002018-02-15T15:23:05-05:00Freckle 500 errors<p><small>Feb 15, 15:23 EST</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Feb 15, 15:17 EST</small><br><strong>Identified</strong> - Unfortunately our hosting provider is doing further emergency maintenance which affects connectivity to our main database, which in turn causes 500 errors in the web app.</p><p><small>Feb 15, 15:07 EST</small><br><strong>Investigating</strong> - We're seeing 500 errors and are investigating.</p>tag:status.letsfreckle.com,2005:Incident/16049622018-02-15T14:47:15-05:002018-02-15T14:47:15-05:00Freckle is down<p><small>Feb 15, 14:47 EST</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Feb 15, 13:18 EST</small><br><strong>Monitoring</strong> - Our hosting provider has finished maintenance. Freckle is all up and running, we will monitor.</p><p><small>Feb 15, 13:08 EST</small><br><strong>Identified</strong> - Freckle is down due to emergency hardware maintenance of our hosting provider. We will post updates as we receive them.</p><p><small>Feb 15, 13:04 EST</small><br><strong>Investigating</strong> - Freckle is currently down, we're investigating.</p>tag:status.letsfreckle.com,2005:Incident/16017022018-02-13T13:05:21-05:002018-02-13T13:05:21-05:00Infrastructure Updates<p><small>Feb 13, 13:05 EST</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Feb 13, 13:02 EST</small><br><strong>Monitoring</strong> - Freckle's server infrastructure will be going through some upgrades. Freckle should be generally available, but will be offline for a short period of time (up to 10 minutes)</p>tag:status.letsfreckle.com,2005:Incident/15629102018-01-13T10:06:53-05:002018-01-13T10:06:53-05:00Server Software Updates<p><small>Jan 13, 10:06 EST</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Jan 13, 09:31 EST</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Jan 11, 16:05 EST</small><br><strong>Scheduled</strong> - We're updating software on our servers, and due to required reboots Freckle will not be available for a short period of time during the maintenance window. We expect a downtime of less than a minute.</p>tag:status.letsfreckle.com,2005:Incident/13149642017-08-05T16:00:28-04:002017-08-05T16:00:29-04:00Infrastructure Updates<p><small>Aug 5, 16:00 EDT</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Aug 5, 14:00 EDT</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Aug 2, 13:26 EDT</small><br><strong>Scheduled</strong> - Freckle's server infrastructure will be going through some upgrades. Freckle should be generally available, but may be offline for short periods of time (up to 10 minutes at a time)</p>tag:status.letsfreckle.com,2005:Incident/12705572017-06-19T18:11:25-04:002017-06-19T18:11:26-04:00Slow loading and timeouts<p><small>Jun 19, 18:11 EDT</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Jun 19, 17:58 EDT</small><br><strong>Monitoring</strong> - A fix has been implemented and we are monitoring the results.</p><p><small>Jun 19, 16:36 EDT</small><br><strong>Investigating</strong> - We're looking into Freckle being very slow to load or timing out completely.</p>tag:status.letsfreckle.com,2005:Incident/12662462017-06-14T13:48:08-04:002017-06-14T13:48:08-04:00Slow loading and timeouts<p><small>Jun 14, 13:48 EDT</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Jun 14, 12:43 EDT</small><br><strong>Monitoring</strong> - We've implemented a fix to improve Freckle's overall performance, and are monitoring the results.</p><p><small>Jun 14, 11:44 EDT</small><br><strong>Investigating</strong> - We're looking into Freckle being very slow to load or timing out completely.</p>tag:status.letsfreckle.com,2005:Incident/12248562017-05-20T12:00:15-04:002017-05-20T12:00:15-04:00Infrastructure Updates<p><small>May 20, 12:00 EDT</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>May 20, 10:00 EDT</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>May 4, 11:44 EDT</small><br><strong>Scheduled</strong> - Freckle's server infrastructure will be going through some upgrades. Freckle should be generally available, but may be offline for short periods of time (up to 10 minutes at a time)</p>tag:status.letsfreckle.com,2005:Incident/11862962017-04-08T00:45:04-04:002017-04-08T00:45:04-04:00Infrastructure Updates<p><small>Apr 8, 00:45 EDT</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Apr 7, 23:45 EDT</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Apr 4, 11:28 EDT</small><br><strong>Scheduled</strong> - Freckle's server infrastructure will be going through some upgrades. Freckle should be generally available, but may be offline for short periods of time (up to 15 minutes at a time)</p>tag:status.letsfreckle.com,2005:Incident/11867722017-04-05T00:45:22-04:002017-04-05T00:45:23-04:00Freckle is down and not loading<p><small>Apr 5, 00:45 EDT</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Apr 4, 21:19 EDT</small><br><strong>Update</strong> - Our hosting provider has resolved the underlying issue. We'll keep monitoring just in case.</p><p><small>Apr 4, 20:46 EDT</small><br><strong>Monitoring</strong> - Our hosting provider still shows the outage, but our monitoring systems show Freckle as online and our engineering team is actively monitoring Freckle's status. Thanks to everyone for your patience as we've been dealing with this outage, and we're continuing to monitor Freckle (as always!)</p><p><small>Apr 4, 20:30 EDT</small><br><strong>Update</strong> - We're still waiting for an update from our hosting provider, but our monitoring tools show the web servers are online and accepting requests. However, the database is still in a degraded state and we're waiting for it to return to normal/actively monitoring it.</p><p><small>Apr 4, 19:09 EDT</small><br><strong>Update</strong> - We're still waiting for an update from our hosting provider, and will update our status page as soon as possible.</p><p><small>Apr 4, 18:48 EDT</small><br><strong>Update</strong> - Our hosting provider's engineering team is continuing to work on fixing the issue. We'll update here as soon as we get more information.</p><p><small>Apr 4, 18:32 EDT</small><br><strong>Identified</strong> - We've been informed by our hosting provider that there's an issue with their hosting environment and they're working on resolving it. We'll keep posting updates here.</p><p><small>Apr 4, 18:11 EDT</small><br><strong>Investigating</strong> - Freckle is down and not loading. We're identifying the root cause right now and will keep you posted.</p>tag:status.letsfreckle.com,2005:Incident/11707052017-03-25T17:00:24-04:002017-03-25T17:00:25-04:00Infrastructure upgrades and DNS change<p><small>Mar 25, 17:00 EDT</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Mar 25, 16:59 EDT</small><br><strong>Verifying</strong> - Freckle is back online! Because of a DNS change, it may take a few hours for you to work from your location; however for most customers (when their DNS servers are configured correctly) Freckle should be back and reachable! We're monitoring the new server setup and will keep posting updates here.</p><p><small>Mar 25, 16:39 EDT</small><br><strong>Update</strong> - Hosting upgrades are complete; we're now testing the new Freckle infrastructure and anticipate being back online shortly after completing our tests.</p><p><small>Mar 25, 16:21 EDT</small><br><strong>Update</strong> - Hosting upgrades are proceeding and we're anticipating being back online within the next hour.</p><p><small>Mar 25, 15:29 EDT</small><br><strong>Update</strong> - Freckle will now go offline for maintenance. We anticipate Freckle being back online in a few hours and keep posting updates here.</p><p><small>Mar 25, 14:00 EDT</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Mar 24, 16:41 EDT</small><br><strong>Scheduled</strong> - Freckle will be unavailable because of hosting upgrades. This upgrade also involves a DNS change, which, while the upgrade itself should be over in 1-2 hours, might take several hours to propagate to all customers. We expect that the scheduled maintenance will last 1-2 hours for most customers.</p>tag:status.letsfreckle.com,2005:Incident/11545602017-03-10T16:41:00-05:002017-03-10T16:42:35-05:00Freckle is down<p><small>Mar 10, 16:41 EST</small><br><strong>Resolved</strong> - We've been given the A-OK from our hosting provider that the hardware issue has been fixed. All systems are working normally now. We'll of course continue to monitor (as we always do!). Happy weekend!</p><p><small>Mar 10, 16:22 EST</small><br><strong>Monitoring</strong> - And Freckle is back! Yay! We'll keep monitoring and post updates as needed.</p><p><small>Mar 10, 16:19 EST</small><br><strong>Update</strong> - Our hosting provider are right now replacing defective hardware on the web host—we'll keep you posted on the progress and update here as soon as we know more (and hopefully have an ETA for when Freckle will be back up soon!).</p><p><small>Mar 10, 16:00 EST</small><br><strong>Update</strong> - Our hosting provider's engineering team is continuing to work on fixing the issue. We'll update here as soon as we get more information.</p><p><small>Mar 10, 15:53 EST</small><br><strong>Update</strong> - We're in contact with our hosting provider's engineering team and working to identify and resolve the problem. The issue specifically affects our web servers only; the database/backend is functioning normally (but can't be accessed without the web front-end).</p><p><small>Mar 10, 15:42 EST</small><br><strong>Identified</strong> - We've been informed by our hosting provider that there's an issue with their hosting environment and they're working on resolving it. We'll keep posting updates here.</p><p><small>Mar 10, 15:40 EST</small><br><strong>Investigating</strong> - We just got alerted by our monitoring that Freckle is down and we're investigating.</p>tag:status.letsfreckle.com,2005:Incident/11427182017-03-05T14:00:24-05:002017-03-05T14:00:24-05:00Infrastructure Updates<p><small>Mar 5, 14:00 EST</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Mar 5, 11:00 EST</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Feb 28, 15:49 EST</small><br><strong>Scheduled</strong> - Freckle's server infrastructure will be going through some upgrades. Freckle should be generally available, but may be offline for short periods of time (up to 10 minutes at a time)</p>tag:status.letsfreckle.com,2005:Incident/11419332017-02-28T13:44:19-05:002017-02-28T13:44:19-05:00PDF Downloads not working<p><small>Feb 28, 13:44 EST</small><br><strong>Resolved</strong> - PDF generation is working again! We'll be keeping an eye on it, though!</p><p><small>Feb 28, 13:18 EST</small><br><strong>Identified</strong> - Due to an outage with DocRaptor, the 3rd party service we use to generate PDFs, PDF exports of reports and invoices are unavailable. We are actively monitoring the 3rd party service and will notify you when the outage is over.</p>tag:status.letsfreckle.com,2005:Incident/10056542016-12-01T10:48:18-05:002016-12-01T10:48:19-05:00Report downloads are not working<p><small>Dec 1, 10:48 EST</small><br><strong>Resolved</strong> - Everything is back to normal! If you still have any issues downloading reports or think you're seeing out of date statistics, please let us know at support@letsfreckle.com so we can have a look. Thanks for your patience while we fixed this—we know the first day of the month is a busy day for invoicing and payroll and it's not helpful when your time tracking software is having trouble generating the Excel files and PDFs you need. :(</p><p><small>Dec 1, 10:21 EST</small><br><strong>Update</strong> - We're deploying an update that will fix the underlying cause of the issue and will allow our background processing to catch up on report statistics and make sure all those numbers you see on the reports page are the latest and greatest. We'll report back once everything is back to normal.</p><p><small>Dec 1, 10:08 EST</small><br><strong>Update</strong> - A secondary impact from the earlier issue is that summaries of reports on the reports page (e.g. total sum, etc.) are in some cases slightly out of date. The sums will be correct if you run the report. We're working on getting this up-to-date again.</p><p><small>Dec 1, 09:28 EST</small><br><strong>Monitoring</strong> - Download are working again. Yay! We're monitoring the fix and the server performance and will update this incident as needed.</p><p><small>Dec 1, 09:20 EST</small><br><strong>Identified</strong> - Downloads are not working right now. We've identified the problem and are working on bringing everything back up.</p>tag:status.letsfreckle.com,2005:Incident/9788352016-11-06T09:48:30-05:002016-11-06T09:48:30-05:00Server software updates<p><small>Nov 6, 09:48 EST</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Nov 6, 09:45 EST</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Nov 4, 08:58 EDT</small><br><strong>Scheduled</strong> - Due to software updates on our servers, Freckle will be unavailable for a short period of time (we anticipate less than a minute) during the maintenance window.</p>tag:status.letsfreckle.com,2005:Incident/9510292016-10-17T10:03:58-04:002016-10-17T10:03:58-04:00Outage due to configuration issue<p><small>Oct 17, 10:03 EDT</small><br><strong>Resolved</strong> - Freckle was unavailable for about a minute today shortly before 10am EST. This was due to a configuration issue that has been resolved. Sorry about this! All systems are go now!</p>tag:status.letsfreckle.com,2005:Incident/7644332016-09-18T08:29:28-04:002016-09-18T08:29:29-04:00Server software updates<p><small>Sep 18, 08:29 EDT</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Sep 18, 08:02 EDT</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Sep 11, 12:02 EDT</small><br><strong>Scheduled</strong> - We're updating software on our servers which will require server restarts. This will lead to a short period of downtime (less than a minute) during the maintenance window.</p>tag:status.letsfreckle.com,2005:Incident/7487542016-09-09T07:01:07-04:002016-09-09T07:01:08-04:00Datacenter network maintenance<p><small>Sep 9, 07:01 EDT</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Sep 9, 03:00 EDT</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Aug 25, 18:32 EDT</small><br><strong>Scheduled</strong> - Our data center is upgrading its network infrastructure, which will lead to about 5 minutes of downtime during the maintenance window.</p>tag:status.letsfreckle.com,2005:Incident/5285022016-06-13T01:34:44-04:002016-06-13T01:34:45-04:00Freckle is down and not loading<p><small>Jun 13, 01:34 EDT</small><br><strong>Resolved</strong> - Freckle is back up after our hosting provider experienced a temporary network outage. Our ops team will continue to monitor the app.</p><p><small>Jun 13, 01:11 EDT</small><br><strong>Monitoring</strong> - Freckle is back up, but we're continuing to investigate the root cause and monitor the app.</p><p><small>Jun 13, 01:00 EDT</small><br><strong>Investigating</strong> - Freckle is down and not loading. We're identifying the root cause right now and will keep you posted.</p>tag:status.letsfreckle.com,2005:Incident/5271662016-06-11T09:49:13-04:002016-06-11T09:49:14-04:00Server software updates<p><small>Jun 11, 09:49 EDT</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Jun 11, 09:30 EDT</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Jun 10, 08:16 EDT</small><br><strong>Scheduled</strong> - We're performing routine software updates on our servers. Due to necessary reboots, Freckle will be unavailable for a short period of time (we expect less than a minute) at some point during the maintenance window.</p>tag:status.letsfreckle.com,2005:Incident/4803772016-05-01T17:33:45-04:002016-05-01T17:33:46-04:00Server Software Updates<p><small>May 1, 17:33 EDT</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>May 1, 12:00 EDT</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Apr 26, 12:33 EDT</small><br><strong>Scheduled</strong> - We're updating software on our servers and there may be short periods of time where Freckle is not available.</p>tag:status.letsfreckle.com,2005:Incident/4598062016-04-07T13:33:21-04:002016-04-07T13:33:21-04:00Slow loading and timeouts<p><small>Apr 7, 13:33 EDT</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Apr 7, 12:44 EDT</small><br><strong>Monitoring</strong> - The slowdowns and timeouts appear to have stopped, but we're continuing to monitor Freckle.</p><p><small>Apr 7, 12:30 EDT</small><br><strong>Investigating</strong> - We're looking into Freckle being very slow to load or timing out completely.</p>tag:status.letsfreckle.com,2005:Incident/4490302016-03-24T10:06:00-04:002016-03-24T10:06:00-04:00Increased API latency<p><small>Mar 24, 10:06 EDT</small><br><strong>Resolved</strong> - The underlying reason of API latency was an API client issuing too many requests for large sets of entries. As a temporary fix we've reached out to the affected customer and they've reduced the amount of requests issued. In the longer term, we're working on better caching for the more popular API methods; this will all but eliminate latency issues like this one.</p><p><small>Mar 24, 06:55 EDT</small><br><strong>Investigating</strong> - We're seeing some slow-downs in processing API requests. The API is still functioning fully, but response times are slow for some types of API calls.</p>tag:status.letsfreckle.com,2005:Incident/4490272016-03-24T06:46:18-04:002016-03-24T06:46:18-04:00Background processing delays<p><small>Mar 24, 06:46 EDT</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Mar 24, 06:35 EDT</small><br><strong>Monitoring</strong> - After our hosting provider's scheduled maintenance some network connections weren't restored correctly leading to background processing queues being stuck. We've restored connectivity and the stuck queues are now being processed normally and are catching up.</p>