Recent Server Outage

The following is an explanation from the data-center of what occurred over the weekend, and why some MCGWD customers may have experienced outages.

I would like to fill you in on what happened with your server and what are our options.

We have detected a failed disk drive on SAN5 data storage in our Detroit Cloud unit. In order to prevent any additional issues, we scheduled urgent maintenance.

Maintenance started at 6PM EST, and it was planned to last up to 15 minutes, where all the cloud instances would be put to in-memory paused state, and then unpaused when the maintenance was over.

Unfortunately, during this maintenance, second disk became unstable, which caused instability of the whole SAN unit. At that moment, we have stopped all Cloud instances on that unit, in order to prevent any data corruption, and we have started migrating them to a different SAN units one by one.

Few units, including yours, failed migrating to a different data storage, and we were not able to get them fixed without a serious data loss.

However, I’ve checked backups for your server, and I’ve found that we do have up to date daily backups for every day between December 28th and January 3rd. We still have our system administration team working on this, and we should have more information about it in the morning. How things are standing now, we will most probably be able to rebuild your server and then do a Bare Metal Restore, and avoid any serious data loss.

Sorry for the inconvenience this may have caused. Please give your sites a once over and let me know if anything’s out of whack so that I can get it fixed as quickly as possible.