Reseller giant Misco has confirmed it will embark on a crash migration program to shift its infrastructure into the cloud - a day after a datacentre meltdown that froze its e-commerce front-end for six hours.
The dash to the cloud while having to maintain a creaking inhouse infrastructure illustrates the dilemma facing …

Re: Let me guess

You see the same thing in the cloud as well. People not buying HA configurations and then getting bent out of shape when their non-HA set up is not highly available. Still, kind of a sign of the times when a company that makes their cash selling tin decides to move to the cloud.

Management is not going to blame themselves, 'maybe we should have bought insurance'. It used to be that they would blame the system provider. It isn't always management though. I have seen many a situation where the sys admin hoses up the works and then blames the system for 'crashing.'

DR/BC

Re: DR/BC

In this case, from what they say, RDS would have saved their bacon.

I've done the binary search of RDS snaphot restores to find the corruption time before, and whilst it isn't pleasant it's a lot easier than buying hardware and doing lots of restores and txn log replays.

Re: DR/BC

Even with the cloud, a lot of times the problem is that management knocks off the HA line items to lower their cloud cost. If you use Geo stretch clusters, it is pretty hard to knock down, e.g. Google search, Bing search is never down. It costs serious money though.

Datacentre - or Broom closet?

The more I read about the Misco outage the more I wonder just what "Datacentre" means in their case.

Picture a standard bit-barn - usually a prefab building the size of a soccer field:

you walk in the front door , and (if it's any decent bit-barn) you have to go through various physical security checks.

Having passed them, you go through the security gate / airlock into the data-floor, which may be divided into separate halls, or for our purposes is just a single massive area.

Off in the distance, in the middle of an otherwise bare floor, stands a single 42U rack, and as we get closer, we can see that it's partially populated.

Close up, we see a firewall, a switch, a few ethernet cables and a 2U server with "Web1" written across the lid in marker pen, under the dust. On the floor of the rack is a box of floppies, marked "backup".

Re: Datacentre - or Broom closet?

<quote>Off in the distance, in the middle of an otherwise bare floor, stands a single 42U rack, and as we get closer, we can see that it's partially populated.decades old, valve operated IBM mainframe in its rusting case.

Close up, we see a firewall, a switch, a few ethernet cables and a 2U serverancient green screen terminals, and old Bell 103 modems with "Web1" written across the lid in marker pen, under the dust. On the floor of the rack is a box of floppiespunch cards, marked "backup".</quote>

Re: Datacentre - or Broom closet?

Those old systems worked well. No such thing as pages erroring out. No eighteen factor authentication. No hour glasses or spinning browsers. No Java. Turn the old, reliable 400 back on. They also went down about once every decade.

Huh. I guess their data center comprises just one server? Or did one server cause the entire infrastructure to become corrupt? Typical management. This guy obviously has no idea how his business is run from a technology standpoint. As noted above, Rush to the cloud, it must be better. Meanwhile, talented, qualified engineers are getting shown the door due to idiots like this guy. I'm sure we'll see an article here on el Reg in 5 minutes about how they are also going to adopt DevOps to solve all their problems...

From bad to worse?

Oops, our backups didn't work. But why work on fixing the problem in our infrastructure that caused it? Instead, let's just sign up to use "someone else's infrastructure" that we don't control or really know much of anything about. Of COURSE that will work so much reliably than anything we control.

Re: From bad to worse?

Absolutely! The Cloud* is brilliant and totally resilient and definitely never has any down time, and if our office loses it's Internet connection we definitely won't have any problem continuing to take orders over the phone using the order system that we can no longer access....

Personally I have one very important first rule for externally hosted cloud solutions. Can your business, or the business segment that relies on that solution, continue to function without it?

If the answer is no then you should keep it in house, and do proper redundancy and data recovery. With the right infrastructure your solution can be just as reliable as the cloud, and if you lose net access you can still see it even if your customers can't. Also just as importantly, if it does go wrong you have full control over fixing it instead of twiddling your thumbs waiting for an explanation and/or estimated fix time.

*WTF does that really mean anyway?!? The Cloud? So there's just one right? AWS, Azure etc are all just one big service. Arrgh, don't get me started on that part.

PFI?

Outsourcing data centres is just like PFI - it takes the cost out of the capital budget and removes the provision for ongoing maintenance.

All you have is payment from the current account for the service.

It may turn out to cost you more, it may turn out to be less secure efficient and reliable, but meanwhile you have maximised value, sold off a capital asset, reduced headcount and returned cash to the shareholders.

Future problems? Blame goes straight outside the company to the service provider.

Misco management should take careful note of the issues 123-reg customers are currently facing. Shifting server hosting to a cloud provider (= a computer in someone else's datacentre) might be a good strategic decision if considered in the cold light of day, but it should not be a knee-jerk reaction to a server crashing and it is definitely not an alternative to good service continuity planning (and testing).