Posts Tagged 'Challenges'

We hosted our first quarterly networking event in Singapore yesterday, and as I spoke with the partners, prospects, customers and SLayers in attendance, I heard some incredible stories about struggles with scaling infrastructure and how SoftLayer has revolutionized the way our customers look at their physical and virtual infrastructure. As we talked about our experiences, one of my own "war stories" came to mind, and I got to share it:

On on a Sunday afternoon in March 2002, an earthquake hit Taiwan. It measured 6.8 on the Richter scale, and it shook buildings across the island, flattening some of them and wreaking general havoc in cities. Beyond the visible damage it caused, it took out the fiber landing stations in Taiwan, cutting off Asia Pacific Internet traffic from the US. Typically when a fiber cable system is cut, telcos will scramble to re-route their traffic to the next available path, but because North Asia was crippled by the quake, all Internet traffic in Asia was being routed through Australia, causing major congestion down under, resulting in virtually zero Internet connectivity to the rest of the world.

At that time, I was VP of Sales for a leading Singapore-based hosting company. I received a call on my sales hotline at 7am on the morning after the earthquake. The caller was the CEO of a major gaming company in Australia, and he sounded desperate. All his servers — hosted in the US at the time — were unreachable, and he had been calling hosting companies all over Asia to buy some dedicated servers to host the game for his Asian customers. While I couldn't help him when it came to getting connectivity to his servers in the US, I thought it would be easy to accommodate his request for hardware based in Asia.

I asked him what server configurations he needed, and he detailed 20 identical servers that needed to be up and running for his gaming application within 24 hours, highlighting that he was losing thousands in revenue by the day. He explained that the projected revenue loss would exponentially increase to thousands per hour if the game remained offline for 24 hours more. He gave me his RAM, hard disk, OS and Database requirements, and he added, "We need all of them to be on Woodcrest!"

I remember vividly saying, "Woodcrest what? Oh, yes, yes, we have those!" I told him I'd get back to him, hung up the phone and went straight to our provisioning manager. We stock to provide 20 servers, but we didn't have any Woodcrest CPUs. There was no way we could locate, rack and provision the requested servers 24 hours ... The best we could commit to was 10 days. Obviously, that wasn't going to work, but I wasn't discouraged. I was going to solve the problem.

I managed to scrape together 20 Woodcrest CPUs from different local electronics retailers, and after wrangling cheques from the finance department and getting the CEO to apply pressure the provisioning manager, I was able to "fast-track" the servers to a four-day provisioning time. When all was said and done, he was able to bring his game back online after losing out on 8 days of business. Despite the losses, being able to turn around that kind of order that "quickly" made me pretty proud.

10 years later, I can't believe how much things have changed.

SoftLayer automates almost all of the manual processes, and we're able to provision a dedicated servers in 2-4 hours. While that's a pretty impressive feat, it's even more amazing when you consider that we can bring up 20, 50 or 100 dedicated servers in the same time frame. Just look at what OMGPOP was able to do when their "Draw Something" app was downloaded 36 million times. That's what automation is all about. Anything that we can automate, we automate, and that makes for an unbeatable user experience.

If someone came to us today with the an urgent order similar to the one I dealt with in 2002, the entire interaction above would boil down to, "What specs do you need? *typing* Here's your order number. You can expect the machines to be provisioned within 4 hours." We'd be off the phone by about 7:20am, and by noon, all of the servers would be online and hosting the game. The craziest part is that we're not even satisfied with that turnaround time yet. Our commitment is to continue to innovate, automate and empower our customers through our customer portal and APIs, and because our goals are to get better and serve our customers faster, the carrot will always be in front of us ... the same way UPS has a philosophy of "constructive dissatisfaction."

I want to thank everyone who came to our networking event yesterday. I hope you learned a little something about SoftLayer because I certainly learned a lot about our customers in the dozens of conversations I had. If you weren't able to attend and want to see what you missed, we posted a few pictures on Flickr: SoftLayer Singapore - Quarterly Networking Event - March 28, 2012

Do you have any infrastructure horror stories from the past like mine?

Not to toot my own horn, but I am an actual Rocket Scientist (well an Aerospace Engineer, but Rocket Scientist sounds way cooler). When you are a Rocket Scientist, most of your time is spent in dealing with facts – universal constants, formulas, and a data set that has been validated countless times over. My role at the CTO at SoftLayer is sometimes a challenge because I have to deal with the terrific hyperbole that the tech world inevitably creates. Consider the Segway, Unified Messaging, etc. I think that cloud computing has also fallen prey.

The cloud promises a lot and it does deliver a lot.

Control puts decisions and actions in the hands of the customer. Self-service interfaces enable automated infrastructure provisioning, monitoring, and management. APIs provide even greater automation by supporting integration with other tools and processes, and enabling applications to self-manage.

Flexibility provides a broader range of capabilities and choices, enabling the customer to strike ideal balance of capital and operating expenses. In addition, access to additional infrastructure resources happens in minutes rather than week enabling you to respond "on demand" to changes in demand.

Flexibility and control combined give administrators more choice. Who manages infrastructure (Internal staff or service provider?) Where are workloads processed in an internal datacenter or in the public cloud? When are workloads processed – is this resource-driven or priority-driven? How much to consume – is this policy-driven or demand-driven? How is IT consumed – via central administration or self-service?

Despite its numerous benefits, the operational and cost effectiveness for many enterprises is challenged by the fact that most cloud services come in limited configurations and only serve as standalone solutions. One cloud does not fit all – Fixed specs do not allow administrators to optimize a cloud environment with the ratio of processing power, memory or storage that its intended application needs for its best performance. Most cloud service providers offer a relatively small number of preconfigured virtual machine images (VMI), often starting with small, medium and large VMIs, each with preset amounts of CPU, RAM and storage. The challenge is that even a few sizes (versus only one) don't fit everybody's needs. For example, applications perform best when they are running on servers with optimized configurations. And every application has unique resource demands. If the server is "too small," performance issues may arise. If the server is "too large," the customer ends up paying for more resources than necessary.

To a degree we have already been doing lots of "cloudy" things given our focus on automation. Combine that with a set of tools that let customers self-provision and I think you see where I am headed. The next step up the value chain is SoftLayer's "Build Your Own Cloud" solution. It delivers all of the benefits that I discussed above, but adds the logical step of handing configuration control to the customer. Customers are able to determine a number of things about the environment that their cloud sits on.

The end result is a cloud environment that is fit for customer purpose and customer cost. A classic win-win situation. I wonder what we will think of next.

I have a friend who recently took a CTO position with a medium-sized company. The huge company that previously employed him moved their entire IT staff a long way outside of Texas to a rather unpleasant location as a cost cutting move. He and many others declined the relocation offer. I can’t say as I blame them.

The other day, he told me some of the interesting things he’s found at his new company. This company is not a technology company but a professional services company. Up to now, they have opted to be in the IT business by running their own data center. To keep this post to a reasonable length, I’ll just mention a few of the things he’s run into.

Keeping the room powered and cool – trust me, this is harder than it sounds. It involves things like redundant power, UPS devices, generators, CRAC units, dehumidifiers, fire suppression, etc. All this stuff must be tested and maintained constantly.

Ordering new servers – they have to go through an online configurator, and then wait to receive the shipment. Once it arrives, they have to unpack it, rack it, power it up, and install the software. The cycle time from ordering a new server to getting it into production can stretch from days to weeks.

Tracking assets – needless to say, he’s found several holes in the process here. Knowing how much RAM is supposed to be in each server vs. what’s really there is a struggle. Heck, even knowing what servers are supposed to be there is a challenge. It seems that as servers are moved, replaced, or disposed of that the asset tracking system and processes are not as solid as he would like. These loose operations also bring heat from accountants and auditors, especially if a server ‘s value is still on the balance sheet but it has actually been tossed out and they no longer own it.

Maintenance – they pay for a service agreement where a tech is guaranteed to be onsite in 4 hours to do anything up to a complete rip and replace to get them back in production. Once he asked why several servers, each north of $10,000 in value, were just laying around in a parts cage. He was told these were for spare parts in case of an emergency, just in case they couldn’t wait 4 hours.

Bankers and lessors routinely ask us who our biggest competitors are. We routinely tell them that they are not other hosting companies – they are companies like the one described above that insist on being in the data center business even though they are not IT infrastructure companies. Since these companies are our largest competition, let’s look at how SoftLayer beats the competition on the items listed above.

Keeping the room powered and cool – as a customer of SoftLayer, you simply don’t have to worry about all this. Not at all. This is a huge savings of time, effort, and money.

Ordering new servers – Once you either run through the configurator or call your SLales rep with your order, your new servers are immediately provisioned. The cloud products are up in minutes, and you can have a few HUNDRED dedicated servers ready for production in a few hours. Not in days or weeks or months.

Tracking assets – From the accounting side of things, you just don’t have to worry about tracking the assets at all as a SoftLayer customer. They are an operating expense to our customers, not a capital expenditure. As far as knowing what assets you have to work with, you have access to the best customer portal in the business where every detail about every server is kept up in real time, right down to the individual sticks of RAM and drive configurations of each server. If you need tighter integration, SoftLayer provides an API to put all this information seamlessly into your environment. Disposing of a server is a simple cancellation ticket. It couldn’t be easier.

Maintenance – this is also a simple ticket submission, which is resolved in an impressive turnaround time. This service is included in SoftLayer’s monthly fees. There is no need to stockpile parts or entire servers for emergencies.

Bottom line, if your business’s core competency is not IT infrastructure, you are being beaten in the IT infrastructure business by SoftLayer. You are spending way too much time, money, and attention to run something that isn’t a part of your business. Hey, if you can’t beat us, then join us!

By the way, my friend is proposing a major project for his company in 2010. That project is getting out of the business of running a datacenter. He faces a lot of resistance to change “the way we’ve always done it” from the other senior executives. From my point of view, it’s a no brainer. But I’m biased I guess. I’d just tell them, hey, don’t run a data center – run your business!

You’ve all heard how great it is to work for an innovative and cutting edge company from other staff, so I won’t bore you with things you already know. Let’s talk SL culture a bit; let’s get into the real “meat” of what motivates and fuels the staff on a day to day basis—the Dallas NOC staff specifically. It’s a bunch of dudes and me, the only chick. Yes, I am brave… very, very brave!

We like to eat, if you didn’t already know. Our own CTO, Sam Fleitman, actually competes in BBQ contests. We are his sponsor; go 3 Bars BBQ (best ribs you will ever eat)! Work hard, play harder, eat like pigs (and eat pigs too, we love bacon), that’s what we know and love! On this particular day, I decided to go pick up Chipotle for anyone who wanted some. Chris “Mank” Menard was goofin’ on William “Blue” Spencer; and, somehow it was decided that if Blue could eat 3 Chipotle burrito’s in under an hour, we’d all chip in and pay for his lunch. Blue can be somewhat of a braggart, so everyone was into it. Little did we know…

He began on his first one, polishing the puppy off in around 3 minutes flat.

Let me be sure you understand and know the Chipotle burrito. These things are packed full: meat, rice, beans, cheese, sour cream, salsa, lettuce; they are a good 1-2 pounds each I’d imagine.

He starts the second one, and this is where we all think he’s going to slow down. That first one was a breeze, but the rice should start to expand in his stomach within no time. Wrong again, this one went down just as easily, but he decided to take a bit more time. This one clocked in at 3 minutes, 30 seconds. Although he was admittedly full, as you can see from his expression, he forged on…

Blue took a brief break, to let it all settle in before the 3rd and final burrito, but only to have some Mt. Dew and mess with our minds. We knew we’d be buying those burritos, especially when he decided to toy with all of us with this pose. Yes folks, that is the 3rd one, almost gone.

The 3rd was downed in about 4 minutes 15 seconds and that was only because he was having too much fun messing around with us, as we stood on the other side of the table in amazement, horror, and total disbelief. The guy is an eating machine!
We even got our next order free, thanks to the Manager at Chipotle (around $150). When I picked up the order, she asked why there were 3 for Blue, I explained the situation; she loved it and told me that if he did it, to give her a call and the next lunch order was on her! So, I kindly gave her a ring; and, the following week, we got lunch paid for completely!

Thanks for entertaining us Blue! What’s next? It’s been long enough…Hopefully you will be reading another blog soon, with another meat-tastic challenge!