tag:www.nameprosstatus.com,2005:/historyNamePros Status - Incident History2018-03-19T14:37:22ZNameProstag:www.nameprosstatus.com,2005:Incident/16336672018-03-08T19:27:33Z2018-03-08T19:27:33ZForum and blog software update<p><small>Mar 8, 19:27 UTC</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Mar 8, 19:16 UTC</small><br><strong>Verifying</strong> - Verification is currently underway for the maintenance items.</p><p><small>Mar 8, 19:03 UTC</small><br><strong>Update</strong> - Prep work done; deploying software update</p><p><small>Mar 8, 19:00 UTC</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Mar 8, 18:55 UTC</small><br><strong>Scheduled</strong> - The forums and blog will be down while we update the underlying software. The downtime is expected to last about 15 minutes.</p>tag:www.nameprosstatus.com,2005:Incident/13918492017-10-02T12:53:14Z2017-10-02T12:53:14ZSearch re-index<p><small>Oct 2, 12:53 UTC</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Oct 2, 12:33 UTC</small><br><strong>Verifying</strong> - Verification is currently underway for the maintenance items.</p><p><small>Oct 2, 09:00 UTC</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Oct 2, 08:51 UTC</small><br><strong>Scheduled</strong> - We're performing a search re-index. This will restore the search functionality that was available prior to the forum software update. Search results will be incomplete until the re-index is finished.</p>tag:www.nameprosstatus.com,2005:Incident/13908592017-10-01T15:54:52Z2017-10-01T15:54:52ZForum software update<p><small>Oct 1, 15:54 UTC</small><br><strong>Completed</strong> - The site is operational. Some search features have intentionally been removed, as they were broken. Conversation search will be returning soon.</p><p><small>Oct 1, 08:03 UTC</small><br><strong>Update</strong> - The forum update is complete. Search results will continue to be incomplete while the search index rebuilds. Please notify us if you notice any bugs that aren't related to search.</p><p><small>Oct 1, 07:43 UTC</small><br><strong>Update</strong> - We're updating search.</p><p><small>Oct 1, 07:38 UTC</small><br><strong>Verifying</strong> - Verification is currently underway for the maintenance items.</p><p><small>Oct 1, 07:16 UTC</small><br><strong>Update</strong> - Deploying the update.</p><p><small>Oct 1, 07:15 UTC</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Oct 1, 06:48 UTC</small><br><strong>Scheduled</strong> - We're updating the software that powers the forums.</p>tag:www.nameprosstatus.com,2005:Incident/13194902017-08-05T21:45:19Z2017-08-05T21:45:19ZDatabase route table update<p><small>Aug 5, 21:45 UTC</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Aug 5, 21:15 UTC</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Aug 5, 21:03 UTC</small><br><strong>Scheduled</strong> - This should be very brief, but the site may be down momentarily during failover.</p>tag:www.nameprosstatus.com,2005:Incident/13052442017-07-22T17:11:47Z2017-07-22T17:11:47ZWebsite down<p><small>Jul 22, 17:11 UTC</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Jul 22, 16:54 UTC</small><br><strong>Monitoring</strong> - The issue has been resolved and the website is back online. There may be brief periods of degraded performance while the upgrade finishes.</p><p><small>Jul 22, 16:47 UTC</small><br><strong>Identified</strong> - Logs for our new backup system took up more storage than expected, and the database system shut down to prevent data loss. We're adding more storage space, at which point the site will be back online. We should be back up in about 15 minutes.</p><p><small>Jul 22, 16:44 UTC</small><br><strong>Investigating</strong> - The website is currently down. We're looking into the cause.</p>tag:www.nameprosstatus.com,2005:Incident/12760832017-06-26T07:29:37Z2017-06-26T07:29:37ZDatabase upgrade<p><small>Jun 26, 07:29 UTC</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Jun 26, 07:25 UTC</small><br><strong>Verifying</strong> - Verification is currently underway for the maintenance items.</p><p><small>Jun 26, 07:23 UTC</small><br><strong>Update</strong> - Downtime beginning now.</p><p><small>Jun 26, 07:17 UTC</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Jun 26, 07:16 UTC</small><br><strong>Scheduled</strong> - The maintenance schedule has been updated.</p><p><small>Jun 26, 07:15 UTC</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Jun 26, 07:12 UTC</small><br><strong>Scheduled</strong> - The website will be unavailable for a second or two as we switch database servers. No data will be lost. If the upgrade fails and we need to revert, the site may be down for up to 30 seconds.</p>tag:www.nameprosstatus.com,2005:Incident/12688942017-06-17T02:30:33Z2017-06-17T02:30:33ZDatabase infrastructure update<p><small>Jun 17, 02:30 UTC</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Jun 17, 01:30 UTC</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Jun 17, 01:27 UTC</small><br><strong>Scheduled</strong> - We're making some improvements to our database infrastructure.</p>tag:www.nameprosstatus.com,2005:Incident/11174152017-02-09T07:16:33Z2017-02-09T07:16:33ZDatabase certificate rotation<p><small>Feb 9, 07:16 UTC</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Feb 9, 07:03 UTC</small><br><strong>Verifying</strong> - Verification is currently underway for the maintenance items.</p><p><small>Feb 9, 06:45 UTC</small><br><strong>In progress</strong> - Database servers are rebooting. Availability should not be significantly affected, as we're doing a rolling update. In the event that the website goes down, it will return within about 5 minutes.</p><p><small>Feb 9, 06:36 UTC</small><br><strong>Scheduled</strong> - Several database servers need their X.509 certificates rotated.</p>tag:www.nameprosstatus.com,2005:Incident/11133482017-02-05T11:30:43Z2017-02-05T11:30:43ZInfrastructure upgrades<p><small>Feb 5, 11:30 UTC</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Feb 5, 11:13 UTC</small><br><strong>Verifying</strong> - Verification is currently underway for the maintenance items.</p><p><small>Feb 5, 07:35 UTC</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Feb 5, 07:16 UTC</small><br><strong>Scheduled</strong> - We'll be upgrading several of our database servers. Downtime is expected to be 2-5 minutes, but could last longer.</p>tag:www.nameprosstatus.com,2005:Incident/11020582017-01-25T08:12:37Z2017-01-25T08:15:50ZInfrastructure upgrades<p><small>Jan 25, 08:15 UTC</small><br><strong>Postmortem</strong> - The latest update put too much I/O strain on newly launched instances, resulting in the first deployment attempt failing to complete in a timely manner. This was resolved by switching the root volume from SSD to HDD, the latter of which has a more consistent I/O guarantee for small volumes.</p><p><small>Jan 25, 08:12 UTC</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Jan 25, 07:21 UTC</small><br><strong>Verifying</strong> - Web deployment tentatively successful. Waiting for billing and email to come back online.</p><p><small>Jan 25, 07:07 UTC</small><br><strong>Update</strong> - Re-deploying update; all users will be logged out</p><p><small>Jan 25, 06:03 UTC</small><br><strong>Update</strong> - Previous version restored</p><p><small>Jan 25, 05:49 UTC</small><br><strong>Update</strong> - Reverting update: deploy failed</p><p><small>Jan 25, 05:30 UTC</small><br><strong>In progress</strong> - Deploying update</p><p><small>Jan 25, 05:06 UTC</small><br><strong>Scheduled</strong> - We'll be upgrading several components of our infrastructure tonight. During this time, brief outages may occur.</p>tag:www.nameprosstatus.com,2005:Incident/10630832017-01-01T04:24:02Z2017-01-01T04:24:02ZIntermittent outages<p><small>Jan 1, 04:24 UTC</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Jan 1, 01:50 UTC</small><br><strong>Monitoring</strong> - Service is currently degraded in some regions and fully restored in most. Cloudflare is actively working toward restoring service in regions that are still affected.</p><p><small>Jan 1, 01:32 UTC</small><br><strong>Update</strong> - Cloudflare has resolved the issue in most locations, but the IP addresses for NamePros have changed. You may need to flush your browser's DNS cache or wait a bit before you're able to visit NamePros. Our monitor in Portland, Oregon, United States is indicating that there are still issues around that region.</p><p><small>Jan 1, 01:06 UTC</small><br><strong>Identified</strong> - Cloudflare is indicating that it is experiencing problems near our datacenter.</p><p><small>Jan 1, 01:04 UTC</small><br><strong>Investigating</strong> - We're receiving reports of outages. We suspect these are due to holiday attacks on core internet infrastructure.</p>tag:www.nameprosstatus.com,2005:Incident/9733002016-10-31T04:28:27Z2016-10-31T04:28:27Z502 errors<p><small>Oct 31, 04:28 UTC</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Oct 31, 03:27 UTC</small><br><strong>Monitoring</strong> - An unhealthy server was not automatically taken out of service as it should have been. We've manually resolved the issue and are continuing to monitor for issues. In the future, such problems should be automatically detected and resolved quickly.</p><p><small>Oct 31, 03:17 UTC</small><br><strong>Investigating</strong> - We're looking into high 502 error rates in several locations around the world.</p>tag:www.nameprosstatus.com,2005:Incident/8707012016-09-26T17:11:34Z2016-09-26T17:11:34ZEmail deliverability issues<p><small>Sep 26, 17:11 UTC</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Sep 26, 15:34 UTC</small><br><strong>Monitoring</strong> - Our upstream provider appears to have resolved the issue, but reports that work is still in progress. We'll continue to monitor for errors.</p><p><small>Sep 26, 14:04 UTC</small><br><strong>Identified</strong> - Our outbound email provider has acknowledged the issue and is working to fix it.</p><p><small>Sep 26, 13:43 UTC</small><br><strong>Investigating</strong> - Our outbound email has been experiencing issues since 9:05 AM EDT. We're attempting to establish communication with them.</p>tag:www.nameprosstatus.com,2005:Incident/7211682016-08-01T17:40:37Z2016-08-01T17:40:37ZForum software upgrade<p><small>Aug 1, 17:40 UTC</small><br><strong>Completed</strong> - We've finished the post-upgrade process, though we'll continue to monitor for bugs. A summary of changes will follow soon.</p><p><small>Aug 1, 17:08 UTC</small><br><strong>Update</strong> - A JavaScript error prevented the visual post editor from working for about 15 minutes. It has been fixed.</p><p><small>Aug 1, 16:05 UTC</small><br><strong>Update</strong> - Post-maintenance cache rebuilds have completed. You may find that you've been logged out next time you visit the site.</p><p><small>Aug 1, 08:25 UTC</small><br><strong>Verifying</strong> - The upgrade has completed. Some caches are still being rebuilt, so you may notice incorrect statistics or search results. We're still verifying that the upgrade was successful, so please report any bugs you find.</p><p><small>Aug 1, 05:27 UTC</small><br><strong>Update</strong> - The offline portion of the upgrade is over and the website is back up. If the website looks malformed in your browser, try performing a hard refresh (Ctrl + Shift + R on Windows/Linux, Cmd + Shift + R on Mac).</p><p><small>Aug 1, 04:45 UTC</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Aug 1, 04:30 UTC</small><br><strong>Scheduled</strong> - We're upgrading the forum software on NamePros.com. During this time, the site will be intermittently unavailable. The longest outage will occur around 4:55 AM UTC and is expected to last no more than 15 minutes.</p>tag:www.nameprosstatus.com,2005:Incident/5498232016-07-02T15:30:30Z2016-07-02T15:30:30ZHigh error rate<p><small>Jul 2, 15:30 UTC</small><br><strong>Resolved</strong> - The issue has been resolved.</p><p><small>Jul 2, 15:25 UTC</small><br><strong>Update</strong> - The issue appears to be resolved. We'll continue to monitor the situation.</p><p><small>Jul 2, 15:25 UTC</small><br><strong>Monitoring</strong> - The servers have been rotated.</p><p><small>Jul 2, 15:22 UTC</small><br><strong>Identified</strong> - PHP crashed on one web server. Normally this results in the server being taken out of service automatically, but that didn't happen in this case.</p><p><small>Jul 2, 15:18 UTC</small><br><strong>Investigating</strong> - We're investigating reports of errors.</p>tag:www.nameprosstatus.com,2005:Incident/5199142016-06-01T04:03:24Z2016-06-01T04:03:24Z502 errors<p><small>Jun 1, 04:03 UTC</small><br><strong>Resolved</strong> - The operation has completed without further problems.</p><p><small>Jun 1, 03:39 UTC</small><br><strong>Update</strong> - We need to reattempt the operation that caused the server to crash. The site may go down again, but we will be monitoring it.</p><p><small>Jun 1, 03:27 UTC</small><br><strong>Monitoring</strong> - A fix has been implemented.</p><p><small>Jun 1, 03:20 UTC</small><br><strong>Identified</strong> - An application server crashed and wasn't automatically taken out of service.</p><p><small>Jun 1, 03:17 UTC</small><br><strong>Investigating</strong> - We're looking into 502 errors.</p>tag:www.nameprosstatus.com,2005:Incident/5142472016-05-24T07:28:05Z2016-05-24T07:28:05ZSlow page loading<p><small>May 24, 07:28 UTC</small><br><strong>Resolved</strong> - The initial route leak that we detected has been resolved. Global monitors have been stable for 30 minutes. For further updates on the incident from CloudFlare: https://www.cloudflarestatus.com/incidents/n6rfp29dy464</p><p><small>May 24, 07:11 UTC</small><br><strong>Update</strong> - Log analysis shows that select parts of Europe and APAC were also affected by the routing issue.</p><p><small>May 24, 06:59 UTC</small><br><strong>Monitoring</strong> - Many requests from the US were incorrectly being routed to CloudFlare's Warsaw PoP. The issue has tentatively been resolved, but we're contacting CloudFlare to confirm that there won't be further problems.</p><p><small>May 24, 06:47 UTC</small><br><strong>Update</strong> - There appears to be a connectivity issue between CloudFlare and our primary datacenter.</p><p><small>May 24, 06:44 UTC</small><br><strong>Investigating</strong> - We're currently investigating increased page load times.</p>tag:www.nameprosstatus.com,2005:Incident/5062122016-05-17T02:33:06Z2016-05-17T02:33:06ZNetwork issues near Dallas, US<p><small>May 17, 02:33 UTC</small><br><strong>Resolved</strong> - No further issues detected.</p><p><small>May 17, 02:13 UTC</small><br><strong>Monitoring</strong> - We're waiting to see whether the problem continues before contacting CloudFlare.</p><p><small>May 17, 02:07 UTC</small><br><strong>Investigating</strong> - We're looking into reports of connection failures from our monitors in Dallas.</p>