tag:modmore.statuspage.io,2005:/historymodmore.com Status - Incident History2017-08-17T10:05:00+02:00modmore.comtag:modmore.statuspage.io,2005:Incident/12222322017-05-02T20:55:17+02:002017-05-02T20:55:18+02:00Server Upgrade<p><small>May 2, 20:55 CEST</small><br><strong>Resolved</strong> - And we're back online!</p><p><small>May 2, 20:51 CEST</small><br><strong>Update</strong> - The upgrade is now in progress.</p><p><small>May 2, 20:38 CEST</small><br><strong>Monitoring</strong> - In about 15 minutes the site will go down for up to 10 minutes to install some additional hardware, all services except our forum will be unavailable for up to 10 minutes while this happens.</p>tag:modmore.statuspage.io,2005:Incident/11875332017-04-05T15:42:47+02:002017-04-05T15:42:48+02:00Forum DNS misconfiguration<p><small>Apr 5, 15:42 CEST</small><br><strong>Resolved</strong> - Due to a DNS misconfiguration, the forum may have been inaccessible for approximately 24 hours. The DNS entry has been corrected, and an additional monitoring check has been put in place to alert us sooner of forum issues in the future.</p>tag:modmore.statuspage.io,2005:Incident/11369112017-02-26T10:56:00+01:002017-02-26T10:56:56+01:00Demo site rebuild broken<p><small>Feb 26, 10:56 CET</small><br><strong>Resolved</strong> - This has now been resolved. Apologies for the delay.</p><p><small>Feb 24, 12:40 CET</small><br><strong>Identified</strong> - Due to an issue with the demo site rebuild script, it's currently offline. We're working on a fix.</p>tag:modmore.statuspage.io,2005:Incident/11317092017-02-23T01:12:48+01:002017-02-23T01:12:49+01:00Server migration<p><small>Feb 23, 01:12 CET</small><br><strong>Completed</strong> - Server migration complete! Read-only mode has been disabled.</p><p><small>Feb 23, 00:29 CET</small><br><strong>Verifying</strong> - Migration is reported to be complete, we'll now verify everything is working as expected and resolve any issues.</p><p><small>Feb 23, 00:20 CET</small><br><strong>Update</strong> - The transfer should be complete and DNS is propagating. There appear to be some SSL certificate issues, which will be resolved shortly.</p><p><small>Feb 22, 23:55 CET</small><br><strong>Update</strong> - The migration is still in progress.</p><p><small>Feb 22, 22:30 CET</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Feb 22, 16:50 CET</small><br><strong>Update</strong> - Just to clarify, the following will be disabled/read-only during the maintenance window:
<br />
<br />- Account functions (login, editing profile, listing your purchases, API Keys etc)
<br />- Billing (wont be able of adding products to the cart or checking out)
<br />- Package Provider (unable of browsing extras via the package manager, as well as installing, updating, uninstalling etc)
<br />
<br />The website itself will be available, as will the documentation, demo and support sites. The forum will also remain functional, however you will not be able to login unless you already have an active session.</p><p><small>Feb 21, 10:50 CET</small><br><strong>Scheduled</strong> - Wednesday the modmore.com server will be moved to a server with SSDs. No downtime is expected, but we may place the site in read-only/maintenance mode while the migration is ongoing to prevent dataloss. This will also affect package downloads and installations.
<br />
<br />The forum at forum.modmore.com and support via email is not affected by this migration.</p>tag:modmore.statuspage.io,2005:Incident/11050322017-01-27T18:24:08+01:002017-01-27T18:24:08+01:00Forum upgrade in progress<p><small>Jan 27, 18:24 CET</small><br><strong>Resolved</strong> - All done!</p><p><small>Jan 27, 16:40 CET</small><br><strong>Monitoring</strong> - We're upgrading the forums. It may be unavailable for a couple of minutes.</p>tag:modmore.statuspage.io,2005:Incident/10003702016-11-27T13:24:31+01:002016-11-27T13:24:31+01:00Server instability<p><small>Nov 27, 13:24 CET</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Nov 26, 03:42 CET</small><br><strong>Monitoring</strong> - The server is behaving as it should again.</p><p><small>Nov 26, 03:27 CET</small><br><strong>Update</strong> - This affects the main modmore.com, demo site, package provider and background processes. The forum is functional, however hotlinks to some assets on modmore.com which are slowing down page loads at the moment.</p><p><small>Nov 26, 03:25 CET</small><br><strong>Investigating</strong> - Our server seems to be struggling at the moment, showing high load numbers and refusing to connect at times.</p>tag:modmore.statuspage.io,2005:Incident/8247832016-09-23T13:25:55+02:002016-09-23T13:25:55+02:00Forum upgrade underway<p><small>Sep 23, 13:25 CEST</small><br><strong>Resolved</strong> - Forum upgrade complete!</p><p><small>Sep 23, 12:56 CEST</small><br><strong>Monitoring</strong> - We're about to do some maintenance on the forum, which includes a few minutes of downtime.</p>tag:modmore.statuspage.io,2005:Incident/7475142016-08-24T22:34:22+02:002016-08-24T22:34:22+02:00Demo site failing to rebuild<p><small>Aug 24, 22:34 CEST</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Aug 24, 22:31 CEST</small><br><strong>Monitoring</strong> - Demo site is up and running again after a complete rebuild from our development copy.</p><p><small>Aug 24, 22:19 CEST</small><br><strong>Identified</strong> - Our demo site is currently showing errors due to an issue after the rebuild was triggered around 20 minutes ago.</p>tag:modmore.statuspage.io,2005:Incident/4588422016-04-07T14:21:27+02:002016-04-07T14:21:27+02:00High Load<p><small>Apr 7, 14:21 CEST</small><br><strong>Resolved</strong> - It's been behaving properly since this morning.</p><p><small>Apr 7, 11:12 CEST</small><br><strong>Monitoring</strong> - The server is under high load and not responding quickly. Appears to be caused by an issue with the database server, which is back online and load is going down.</p>tag:modmore.statuspage.io,2005:Incident/4473452016-03-22T10:49:45+01:002016-03-22T10:49:45+01:00Server issues<p><small>Mar 22, 10:49 CET</small><br><strong>Resolved</strong> - Looks like we're running normally again, sorry for the troubles. We'll look into what we can do to prevent this from happening in the future.</p><p><small>Mar 22, 10:43 CET</small><br><strong>Identified</strong> - Looks like there's no disk space left, causing (among other things) the database to refuse connections and sessions to fail. Looking into what has been filling it up, while simultaneously creating some breathing room again.</p><p><small>Mar 22, 10:24 CET</small><br><strong>Investigating</strong> - We appear to be having some odd server issues at the moment, we're investigating.</p>tag:modmore.statuspage.io,2005:Incident/4143562016-02-08T13:13:45+01:002016-02-08T13:13:46+01:00Server down<p><small>Feb 8, 13:13 CET</small><br><strong>Resolved</strong> - Server is back up! Looks like a restart kicked it offline for a few minutes.</p><p><small>Feb 8, 13:09 CET</small><br><strong>Update</strong> - Our forum is hosted on separate infrastructure and remains available: https://forum.modmore.com/</p><p><small>Feb 8, 13:08 CET</small><br><strong>Investigating</strong> - Our server appears to be down at the moment. You can still browse most of the site thanks to CloudFlare while we investigate.</p>tag:modmore.statuspage.io,2005:Incident/4138032016-02-07T17:18:11+01:002016-02-07T17:18:11+01:00Forum upgrade<p><small>Feb 7, 17:18 CET</small><br><strong>Resolved</strong> - Forum is updated and back online!</p><p><small>Feb 7, 17:11 CET</small><br><strong>Monitoring</strong> - Expected to take at least a couple more minutes to complete.</p><p><small>Feb 7, 17:09 CET</small><br><strong>Investigating</strong> - We're running an upgrade on the forum, making it temporarily unavailable.</p>tag:modmore.statuspage.io,2005:Incident/3844122015-12-24T10:14:17+01:002015-12-24T10:14:18+01:00Unable to download ContentBlocks<p><small>Dec 24, 10:14 CET</small><br><strong>Resolved</strong> - The problem downloading ContentBlocks has now been resolved! Apologies for the hiccup.</p><p><small>Dec 24, 10:12 CET</small><br><strong>Identified</strong> - The issue has been identified and a fix is being implemented.</p><p><small>Dec 24, 10:02 CET</small><br><strong>Investigating</strong> - It's currently not possible to download ContentBlocks via the provider, we're investigating.</p>tag:modmore.statuspage.io,2005:Incident/3740932015-12-08T23:24:25+01:002015-12-08T23:24:25+01:00Demo site issues<p><small>Dec 8, 23:24 CET</small><br><strong>Resolved</strong> - Should be good now!</p><p><small>Dec 8, 22:39 CET</small><br><strong>Update</strong> - Looking better now. It's still not 100% working as intended, but for the time being it's functional.</p><p><small>Dec 8, 22:24 CET</small><br><strong>Identified</strong> - Earlier we started deploying the new version of the demo site, but it's not entirely going according to plan just yet. The front-end seems to mostly work as expected, but the back-end is pretty much broken.</p>tag:modmore.statuspage.io,2005:Incident/3542022015-11-10T23:45:29+01:002015-11-10T23:45:30+01:00Site down<p><small>Nov 10, 23:45 CET</small><br><strong>Resolved</strong> - Server is back up, service has been restored.</p><p><small>Nov 10, 23:41 CET</small><br><strong>Identified</strong> - The site is experiencing issues currently. Server has restarted, should be up shortly.</p>tag:modmore.statuspage.io,2005:Incident/3209012015-09-23T01:11:25+02:002015-09-23T01:11:25+02:00Glitch in the matrix<p><small>Sep 23, 01:11 CEST</small><br><strong>Resolved</strong> - The temporary outage affected our primary server and was sorted by a reboot. Leaving things in the very capable hands of SkyToaster for the night.</p><p><small>Sep 23, 01:00 CEST</small><br><strong>Monitoring</strong> - For about 10 minutes our server was unreachable, but service has already been restored.</p>tag:modmore.statuspage.io,2005:Incident/3087782015-09-06T18:45:05+02:002015-09-06T18:45:05+02:00Intermittent database connection issues<p><small>Sep 6, 18:45 CEST</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Sep 6, 18:24 CEST</small><br><strong>Monitoring</strong> - Server is back on track, behaving as expected again.</p><p><small>Sep 6, 18:07 CEST</small><br><strong>Identified</strong> - We're seeing high load and intermittent issues connecting to the database causing issues accessing certain pages and features of our site and provider.</p>tag:modmore.statuspage.io,2005:Incident/2695312015-06-26T15:57:39+02:002015-06-26T15:57:39+02:00Intermittent connection issues and sluggish performance<p><small>Jun 26, 15:57 CEST</small><br><strong>Resolved</strong> - We're closing this incident as everything is back online and behaving as expected.
<br />
<br />In the end we lost about 19 hours of data, including purchases and license information, and were down for a total of . We have restored as much as possible and notified clients whose purchases were affected by this data loss.
<br />
<br />There will be a full post mortem on our blog next week, including steps we're taking to prevent data loss and extended outages in the future.</p><p><small>Jun 25, 18:59 CEST</small><br><strong>Monitoring</strong> - We're back online! Continuing to monitor and make sure things are working as expected. Post mortem to follow.</p><p><small>Jun 25, 18:37 CEST</small><br><strong>Update</strong> - Most sites on the failed node have been back at work since this morning, but modmore.com remains inaccessible right now. We're really, really, *really* sorry about the inconvenience and will publish a full post mortem after all of this is over. If the problems on the main server aren't resolved in the next few hours, we'll make a copy available on an unrelated server, potentially on a different domain to get around potential DNS issues.
<br />
<br />If at any time you have any questions about our products or require documentation, please contact us via support@modmore.com or @modmore on twitter. Our support email has remained unaffected.</p><p><small>Jun 25, 10:57 CEST</small><br><strong>Update</strong> - The host has been working through the night to restore affected sites, and still is. modmore.com is still unavailable, but other users such as demo.modmore.com are up and running again. Host is looking into these issues.
<br />
<br />At this point it looks like we've lost up to 24 hours of data, including purchase records and license information. We'll be manually restoring as much of that as possible later today.</p><p><small>Jun 25, 01:31 CEST</small><br><strong>Update</strong> - There have been complications during the emergency maintenance causing the site to be down longer than expected. Host is working with the DC to restore service, which should happen within the next few hours.</p><p><small>Jun 24, 22:52 CEST</small><br><strong>Update</strong> - The server is now down for emergency maintenance. Should be back in < 45 minutes.</p><p><small>Jun 24, 21:45 CEST</small><br><strong>Update</strong> - Our server will be offline for up to 45 minutes between 11pm and 2am CEST for a reboot, to give the datacenter the opportunity to do full hardware diagnostics and to replace parts that need might replacement. Hopefully this will resolve the root issue we've been troubled by over the past day and a half. After the node comes back up it will likely need some more time to fully stabilise.</p><p><small>Jun 24, 02:03 CEST</small><br><strong>Update</strong> - The faulty drive is scheduled to be replaced at around 8am CEST to make sure there's enough hands on deck.</p><p><small>Jun 24, 00:48 CEST</small><br><strong>Identified</strong> - The issue was identified earlier as a failing disk, just waiting for the datacenter to put in a new one which should resolve the issue.</p><p><small>Jun 23, 16:21 CEST</small><br><strong>Investigating</strong> - We're seeing some intermittent issues with our websites, where connections sometimes fail or they are terribly slow to load. This does not seem to affect all users or websites, but it is not yet clear what is the root cause. Our host has been notified and will hopefully resolve this soon.</p>tag:modmore.statuspage.io,2005:Incident/2120672015-05-03T11:59:24+02:002015-05-03T11:59:25+02:00Slow response times<p><small>May 3, 11:59 CEST</small><br><strong>Resolved</strong> - Server's back to being responsive again. Sorry for the interruption.</p><p><small>May 3, 11:21 CEST</small><br><strong>Investigating</strong> - Our server seems incredibly slow to respond right now, with some requests timing out entirely.</p>tag:modmore.statuspage.io,2005:Incident/2007682015-04-15T00:28:13+02:002015-04-15T00:28:13+02:00Enabling CloudFlare on modmore.com<p><small>Apr 15, 00:28 CEST</small><br><strong>Resolved</strong> - Server upgrades have been completed. CloudFlare seems to have propagated without issues.</p><p><small>Apr 14, 23:58 CEST</small><br><strong>Update</strong> - While CloudFlare is propagating, @SkyToaster is also updating some server configuration :-)</p><p><small>Apr 14, 23:46 CEST</small><br><strong>Monitoring</strong> - To provide better performance and a global CDN, we're enabling the awesome CloudFlare. This process should be fairly painless, but it's possible to receive SSL errors and to be unable of connecting temporarily.</p>tag:modmore.statuspage.io,2005:Incident/1935102015-03-30T11:44:18+02:002015-03-30T11:44:18+02:00Server under high load<p><small>Mar 30, 11:44 CEST</small><br><strong>Resolved</strong> - It seems we were randomly targeted by someone with a grudge against the developer of the compromised sites; we were just collateral. Since the developer took action and removed the exploit the number of requests we're receiving has been steadily decreasing. (As the requests were triggered by javascript, all visitors already on the site are still sending off a constant stream of requests, so it's expected we'll continue seeing them until browser caches are updated).
<br />
<br />This is the last update for this issue unless something changes.</p><p><small>Mar 30, 10:42 CEST</small><br><strong>Update</strong> - The requests are still coming in, though not quite as heavy as before and the server is doing fine. We'll continue to monitor.
<br />
<br />The DDOS seems to have originated from compromised sites, which had javascript injected that made every visitor send off a lot of requests to our site. Throw in a whole bunch of visitors to those sites, and you have a nifty distributed attack that was taking up resources. We've worked with the developer of those sites to deal with the problem at the source.
<br />
<br />At this point there's no clue as to why modmore was targeted, or how those compromised sites got compromised in the first place.</p><p><small>Mar 30, 09:52 CEST</small><br><strong>Monitoring</strong> - We've blocked most of what looks like a DDOS attack so our site is up. The traffic is still coming in heavy and seems to be changing patterns from time to time, so we're continuously monitoring.</p><p><small>Mar 30, 09:19 CEST</small><br><strong>Identified</strong> - Turns out we were getting spammed with requests! We've put in place some protection and the server is feeling a lot better now.</p><p><small>Mar 30, 09:04 CEST</small><br><strong>Investigating</strong> - The modmore server is responding (very) slowly, seems to be under high load.</p>tag:modmore.statuspage.io,2005:Incident/1586792015-01-12T00:50:50+01:002015-01-12T00:50:51+01:00Demo site down<p><small>Jan 12, 00:50 CET</small><br><strong>Resolved</strong> - Demo site is back online, apologies for the hiccup.</p><p><small>Jan 12, 00:37 CET</small><br><strong>Identified</strong> - Our demo site seems to have failed to rebuild tonight and is currently inaccessible. Working on a fix.</p>tag:modmore.statuspage.io,2005:Incident/1541772015-01-02T14:38:27+01:002015-01-02T14:38:27+01:00Scheduled Maintenance - In Progress<p><small>Jan 2, 14:38 CET</small><br><strong>Resolved</strong> - We're back online, including the renewed billing engine! Keep an eye out for a blog article with a full announcement of what's new, and let us know (via @modmore on twitter or support@modmore.com) if you notice any issues or have any questions.
<br />
<br />This is the last update on this maintenance item for now. We will continue making small changes over the next couple of days so do send us your feedback.</p><p><small>Jan 2, 14:12 CET</small><br><strong>Monitoring</strong> - The website is back online. Package provider partially up, wont be long until everything's running smoothly again.</p><p><small>Jan 2, 13:34 CET</small><br><strong>Update</strong> - We're starting the deployment to live at last; the site may be inaccessible for some time.</p><p><small>Jan 1, 23:56 CET</small><br><strong>Update</strong> - Behind on schedule a bit now on bringing up the new billing engine. It's a big change that I don't want to rush live so today was spent on making more improvements, fixing bugs and edge cases and further preparations. There's one final blocking issue that will be resolved tonight.
<br />
<br />As it stands now, the actual reconfiguration and deployment of the live site will happen tomorrow (Friday). Expect more updates and temporary downtime then.</p><p><small>Dec 31, 20:48 CET</small><br><strong>Update</strong> - Some last tweaks, fixes and some last minute improvements are done on a staging environment, and we're almost ready to push it all to the live server. This push will require changing server configuration here and there which will affect the rest of the website temporarily. This will likely happen tomorrow and we'll have more updates then.</p><p><small>Dec 31, 13:11 CET</small><br><strong>Update</strong> - The billing engine and related sections of the site (including Account features) are now offline. We'll be back up and running in the new year.</p><p><small>Dec 31, 03:11 CET</small><br><strong>Identified</strong> - In the next 12 hours, we will be disabling the billing and related aspects of the website. This is necessary so we can rewire the website, update to a new version of our bespoke billing engine (featuring a completely rewritten order process and new unlimited license offering) and set it all up with a fresh deployment process based on Gitify that we have been testing.
<br />
<br />Given it's also new years eve, we have planned up to 72 hours to complete this maintenance / upgrade process, though we hope to be back online a lot sooner than that. During the maintenance, it will not be possible to access the Account section of the website or to purchase licenses . Some other areas of the site, including the package provider, may experience various issues for brief periods of time (< 30 minutes) as well.
<br />
<br />We're sorry for the inconvenience of this rather drastic and long maintenance period. Please do reach out to @modmore on twitter or support@modmore.com if you have any questions.</p>tag:modmore.statuspage.io,2005:Incident/1387942014-11-25T02:30:42+01:002014-11-25T02:30:43+01:00Server Down<p><small>Nov 25, 02:30 CET</small><br><strong>Resolved</strong> - We're online and back to normal load again, sorry for the interruption.</p><p><small>Nov 25, 02:25 CET</small><br><strong>Monitoring</strong> - We're back up, though still recovering and sluggish. We expect to be fully responsive in a couple of minutes as things get back in order.</p><p><small>Nov 25, 02:18 CET</small><br><strong>Identified</strong> - Looks like we're a lil' shy on server resources. Give it a few minutes and we'll be back up..</p><p><small>Nov 25, 02:09 CET</small><br><strong>Investigating</strong> - We seem to be having issues with our server at the moment. Looking into it.</p>tag:modmore.statuspage.io,2005:Incident/1307062014-11-01T00:29:27+01:002014-11-01T00:29:27+01:00Demo site is broken<p><small>Nov 1, 00:29 CET</small><br><strong>Resolved</strong> - The demo site is back up and running!</p><p><small>Nov 1, 00:23 CET</small><br><strong>Identified</strong> - The problem seems to have occurred due to a disk space violation while rebuilding the site, something that happens automatically every 24 hours. We're upgrading the virtual user and are triggering a fresh rebuild, which should bring everything back up.</p><p><small>Nov 1, 00:19 CET</small><br><strong>Investigating</strong> - Our demo site seems to be, er, missing some files. We're investigating and working on bringing it back shortly.</p>