Main menu

3-bars-3-questions

Ever since I installed a firefox add-on that would tell me the physical location of servers I visit on the web, I’ve begun noticing that there’s a lot of servers in Los Angeles. Now, at first glance this makes a lot of sense. LA is a sprawling city with millions of people and an enormous telecommunications infrastructure. The real estate is cheap, compared to other cities of similar size, and it’s relatively centrally located as far as the population of the West coast is concerned. I finally realized why it bothered me so much: Earthquakes.

Most tech-savvy users will realize that shaking a computer hard drive during a read or a write could potentially damage your data or even ruin the entire drive. Certain laptop manufacturers (Apple and Lenovo come easily to mind) even have laptops that can sense when they’ve been dropped so they can put an emergency break on the hard drive to prevent this damage. Server hardware doesn’t have that luxury, as servers aren’t generally designed to shake, rattle, or roll.

However, what happens to those LA data centers during an earthquake? Presumably they use the industry-standard solid steel racks on a raised tile floor. I haven’t seen (nor did I come across during a short Google search) any data centers with spring-loaded raised floors. It’s also safe to assume there’s no padding on the servers themselves, as that would exacerbate an already difficult temperature problem all data centers face.

So what happens, do you think? I’ve never worked on a data center on a fault line, but I imagine that, when an earthquake hits, the servers shake just like the rest of the building. And I also imagine that some of those servers are performing reads or writes to their hard drives during that time. I wonder how much data is lost due to earthquakes every year.

At SoftLayer we have intelligently located our main data center thousands of miles from the edge of our tectonic plate, leaving our Dallas customers safely unshaken. Yes, Dallas is at the bottom of Tornado Alley, but that’s where our second genius play comes in. We’ve chosen Dallas instead of Ft. Worth! For those of you not familiar with the DFW metroplex: due to the area’s geography, Ft. Worth receives the majority of the tornado attention for the area, leaving Dallas relatively unscathed.

Even SoftLayer’s coastal data centers are located far from active earthquake zones. Seattle gets far fewer earthquakes than the more Southerly major West coast cities, and our data center in DC hasn’t been shaken in a millennium at least. So to all those companies that put their data in LA-based data centers: why shake your money maker, so to speak?

The following news article came down the SP Wire on Thursday December 18 @ 05:00 Greenwich Mean Time.

At approximately 11:30 Eastern Time senior agent Donald Bolden of the FBI’s cyber crime division held a press conference at the Langley, VA field office to confirm rumors that a laptop containing a large amount of sensitive information had indeed gone missing from the prominent Kringle Corp entity. The information contained on the laptop, is cited as “personal data” by Bolden, but an unnamed source tells this reporter the missing database is purportedly the infamous “naughty list”. Among other items, the stolen laptop holds the full name, address, and social security number for an estimated 2.2 billion minors from all around the world.

“This heinous act of theft truly has the potential to turn this Christmas holiday upside down,” said Bolden. “We urge the responsible party to turn yourself and the laptop in.” At this point in the investigation, the FBI and Kringle Corp are offering amnesty to anyone who brings the stolen property forward. “Don’t kid yourself though,” Bolden warned. “The FBI will find the culprit in this crime and should the laptop not be recovered until after December 24th CEO and founder of Kringle Corp, world renowned philanthropist Christopher Kringle himself, has agreed to prosecute to the full extent of the law.”

When describing exactly how the laptop went missing details get a little fuzzy. According to company logs, the laptop was last signed for by one Edward Keebler. Keebler worked for Elfco, a wholly owned subsidiary of the Kringle Corp located just outside of Mexico City. The laptop was apparently left in Keebler’s car. Company policy strictly prohibits the taking of any company laptop from the premises and as such, Keebler has been let go.

In a statement from Keebler’s lawyer, Mr. Keebler expressed being “truly sorry” for his actions. He states he was aware of the company policy, but under extreme stress from working long hours at the toy assembly plant. He had intended to use the laptop from home to catch up on some paperwork. He now realizes it was a mistake, he agrees with Kringle’s decision to terminate his employment, and he asks nothing more than to be left alone. Keebler’s lawyer also confirmed that his client would be leaving Mexico and returning to his family’s summer home, where he hopes to maybe go back to work as a pastry decorator in his older brother’s cookie factory.

While Bolden states he believes the thief or thieves probably were not aware of what was on the hard drive of the red and white striped laptop when it was taken, the FBI has not yet ruled out that this was an act of sabotage. “Let’s face it,” said Bolden. “Everybody and their brother wants to be Mr. Kringle. If this Christmas gets wrecked, there are a number of other entities out there who stand to gain from Kringle Corp’s misfortune.” Although Bolden refused to name any of Kringle’s competitors specifically, in a public announcement to shareholders just last April, founder and majority stockholder, E. Bunny, of the East-R conglomerate stood before his fellow stake holders and announced his clear intention to move his company into the pole position by the year 2011, using “whatever means necessary”.

While Christopher Kringle himself was not available for comment, his “right-hand-sized” man, Jack Frost, made a statement on his behalf. “Regardless of the outcome to this series of unfortunate events, Mr. Kringle and his entire organization is dedicated to their vision. If we have to double our call center staff in India , telephone every parent, and re-key all the data, then that’s what we’ll do.” Despite the seriousness of this situation, Mr. Frost remained cheerful. As he hurried along with his personal security entourage from the site of the press conference to his corporate sleigh, he took the time to comfort one somber looking little boy and his mother.

“ You can count on us,” Frost told the teary eyed kid. “We are going to do whatever it takes to insure a Merry Christmas to all, and to all a good night!”

You find yourself wondering what Happy SASsy day might be, well I am here to tell you. SoftLayer has done it. We put our processes and procedures on the line and let other people check them out, and we passed. I have never been very good at tests as my brother got 99% of the brain DNA in our family.

That means our SAS 70 assessment is complete, and we are compliant and exception free! We completed it without a single time extension and did it the first time through. What does this mean? This means that our customers are safe with us and our reseller customers are safe as well. It also means they you have been safe with us since day one and now we have been looked at under a microscope to prove it. We have procedures in place to protect our customers and we actually follow them. If you are a public traded company that needs structure, processes, procedures, great sales and support, and a customer portal that is most likely the best and most powerful you have ever seen then now is the time to come give us a try. You have seen the rest now come see the best.

An important goal for most IT shops, at least most that I’ve been in contact with, is to have their software and/or hardware at high availability. Although outages will happen, why would a company want to prolong the outage? I don’t know either, but it does happen. I for one, think processes and procedures are needed to keep an IT shop running smoothly, but when is it too much? Generally when a company has to present to a committee of what the outage is, why it happened, estimated time of repair and whose fault it was.

Personally I’ve seen these meetings take hours!! When all along the problem is a one line code fix! BUT you still have two committees to go through before you can even push the code to production! After the problem has been explained in 14 different ways to 14 different people and a developer has been sacrificed, NOW you can propose your solution and hope they don’t decide to stone you! (Oh yeah, expect about 14 more questions on your solution) Now you get to do the one line code fix, test and build. (This whole step probably took about 10 minutes)

Then you get to present to a committee who will decide if they want to take the responsibility of building/pushing your code or not. Look on the bright side; this meeting won’t take as long as the first meeting and less questions! Once your one line of code has been built/pushed, now you can test – AGAIN, and present your results to the last committee. At last, your code is pushed to production. This whole process can take anywhere from 6 to 24 hours, but the actual fix (the important piece!!!) took less than 10 minutes!! And yes, you the developer get to work the whole time. This is when processes and procedures, or as I like to call it “red tape” get in the way of what is really important – high availability to the customer.

Here at SoftLayer we know what is important to our customers. We also know that high availability is most important to our customers. As a developer, I personally empathize with the customer when sitting during an outage or downtime. We know how crucial it is for us to keep our customers running their business at ALL times, so we keep our processes and procedures streamlined to benefit us developers and most important our customers. We want to stay away from the saying “we need to cut the red tape”. Working here at SoftLayer, I have now seen both sides and believe me when I say “No red tape for me!”.

Remember a goal of 99.99% availability throughout a year does not leave much room for downtime, so use that time wisely. Whew!!! Just be glad it wasn’t a two line code fix.

This weekend I took my son and two of his classmates to a Dallas Stars game. With three 15 year old boys in the car, I simply did the driving and listened to the conversation. Of course the conversation was broader than just the three of them. They all had cell phones, of course, and each one was texting several other friends as they conversed together. So as near as I can tell, the conversation encompassed about a dozen people – three of whom were in my car.

Here’s a snippet of the conversation:

“I hate the auto-correct feature on my iPhone. Whenever I try to type in abbreviations, it changes it to something I don’t want.”

“They fixed that in the latest software update. Here, let me turn that off for you.”

“Cool. Thanks!”

“I like my Blackberry because it has buttons I can actually mash. I can enter text without having to look.”

“What do you think about the new Blackberry Storm? How would you enter text without looking? I mean the screen clicks but there’s no buttons.”

“Hey, why don’t they make a phone where you can enter text with voice recognition? Then you could just speak your words into the phone, they pop up as text on the screen, and the only button you have to push is ‘Send’.”

As SoftLayer prepares its cloud computing offerings for market, I think the same thing is going on. Technology is coming full circle.

When I first started making my living in technology in the late 1980’s, I was a programmer. Anyone remember COBOL? The software company where I worked had a mainframe that they leased from IBM. They didn’t own it. Each employee was connected via terminal – a gargantuan metal monochrome monitor with a clacky metal keyboard that weighed probably over 100 pounds and could withstand a bomb blast complete with a cable that could also be used to dock a cruise ship. The company paid for its IT needs based on CPU time. Naturally, we were motivated to write programs that minimized CPU usage in the mainframe. All of our applications and corporate email were connected to a central IT source.

After the mainframe, standalone desktop machines began popping up. Then they began to be connected with Token Ring or Novell networks and the client/server architecture was born. When the Internet came along, these machines on people’s desks made the phenomena of Cyber Monday possible.

Now according to Lance, everything is “cloud cloud cloud cloud cloud.” SoftLayer’s cloud offering will allow our customers to keep all their applications and corporate email in one central IT source which the customer will not own, i.e., “the cloud.” It will be billed on a usage basis. The more the customer uses, the more he/she will be billed. The computing power can be instantly scaled up or down as needed. Just as I was motivated to be efficient in CPU usage in the 80’s, companies will be able to control costs on the fly by adopting efficient use of cloud resources.

This sounds a lot like the late 1980’s to me. But I never dreamed that that beast of a machine in the room down the hall could ever be called “the cloud.”

You hear a loud whistle in the distance most likely someone’s internet clock set to 5pm. You are finishing up the last 20 or 30 emails you are behind on and getting ready for a little OT. Suddenly, out the window you hear a scream; “Yabba Dabba Doooo!!!!” and you see someone running from the building next door as fast as they can. You realize it is time for all the clock punchers to go home for the evening. Who else would scream “Yabba Dabba Doo”? No one screams it at 5am when the alarm goes off to go to work... at least no one I know, unless it is Starbucks and Krispy Kreme day.

Here at the SL Server Quarry we have some interesting similarities to the Stone Age Flintstones and this blog is intended to point a few of them out. Since a few of them were recently compared to Darth, Luke and their friends I just couldn’t resist. This is my blog so I make the rules, they can be based on looks, personality, or whatever I can come up with that is humorous.
So starting at the top… and the easy ones…

Fred “Lance” SLintstone – The shows main character and the designer of our highly automated quarry. Like Fred Flintstone, he is the task master that gently pushes Barney, and everyone around and makes them meet deadlines that a team of 80 might not be able to meet. These impossible requests keep everyone in the server quarry on the same path and going for the same goal. Rumor has it he is out playing in the clouds most recently.

Barney “Sam” Rubble – Barney is here every morning when I get here and is always here when I leave. In most cases he beats Fred; I think he has faster feet. I have tried many different times of arrival and departure to trip him up but I can’t seem to work longer than he does. I think maybe he just lives under his desk and goes out once a day to move his truck around so people will think he left and came back. That’s why he never parks in the same place. I always thought it was so his employees couldn’t figure out when to show up and leave but I have since foregone that conclusion! Barney keeps Fred SLintstone very happy.

Wilma “JenJen” SLintstone – Lance’s wife of course. Now you would think this would be off limits but she too has quite a role here in the server quarry. She is always ready to help us say things politically correct and get the word out. Unfortunate for her and good for me, PR groups can’t proofread every blog. They just don’t have the time as they have real work to contend with. If they had that much free time then they might want to consider a career change.

Pebbles “Amanda” Slintstone – is one of our SLales gurus. She is one of the best in the industry. I think she could play the part in a movie based on the similarities in the looks alone! If you need a server you can’t go wrong by contacting her.

Mr. “Sean” Slate has a goal. He wants us to have everything we need to make our customers happy. No matter whether it is internal equipment, new products, new toys, whatever, money, cables, pickaxes, sledge hammers, you name it and he helps in that way. Don’t let the smile fool you, there must be a little stress in his position since Mr. Slate is a little light on the topside.

Betty “Sam’s Wife” Rubble – Oh no, I am not falling into this gravel pit. This would be my boss’s wife, I know better!

Bamm Bamm “Michael” Rubble – Bamm Bamm sits quietly in his cube knocking out the sales. He is the pretty boy of the bunch. Even though the only thing he knows how to say is “Bam Bam” it must work in server sales as he does quite well. It is a rumor that he learned the phrase “Bam Bam” while chasing Doug Jackson around with a club. You have to watch that server quarry SLales team.

Dino “Daniel” SLinstone - Dino is fun loving and likes goofing off with Pebbles and Bamm Bamm, he can also be moved to anger, at which point he snarls and snaps. HR currently has Dino in obedience training so we feel certain that he will stop snapping. Rumor has it he makes a mean elephant chili, but I believe it to be exactly that, a rumor.

The “Techies” Gruesomes (Fred’s strange next door neighbors) – We love the Gruesomes. Without them we wouldn’t have the great support we have. They work hard all the time and anyone that knows about some of the things they do must be a little bit strange in a good way. These guys can help you with just about anything you need when it comes to problems in the server quarry. This is a partial pictorial of the industries finest!

Don’t forget The “Information Systems” Wayouts! (The English rock group mistaken for aliens!) – this group of four guys and their team keep all the quarry’s internal brains and lots of quarry customer stuff working as well. Most outsiders know them as humans but all good SLackers know they must be aliens. Who is that smart and human? I can’t tell which group is which, can you?

The Great “Ric” Gazoo (an alien exiled to earth) – The Great Gazoo brings a ton of alien gadgetry to SoftLayer. Without this alien technology we would be just another place on the web. With his alien brain power he makes Lance and Nathan’s ideas come to life.

Arnold “Mike” the Paper Boy – Arnold is the money guy around here but he also writes blogs in hopes that he can keep up with the Kinmans. It will never happen but he can be called the paper boy for those reasons. Not to mention his boyish good looks many, many, many, MANY years ago. We just need to teach him to let go of the flat rocks and chisel and get into the quarry’s more technical side.

Joe “Nathan” Rockhead – Joe is friend of the show. He and Barney go way back. I think they have some kind of collegiate relationship because they say the word Aggie all the time. I’m not quite sure what that is all about. Joe’s head is full of all sorts of things but I am not so sure it is rocks as he keeps us on the cutting edge of everything! I guess it is true; some people do like to read! Joe is kind of an enigma as you can see from his Flintstone comparison picture. (In other words a picture of Joe from the Flintstones just doesn’t exist!)

And last but not least!

Sam ”Kevin – Klaude “my name is not Klaude” Slagheap (Grand Poobah of the Loyal Order of Water Buffaloes) – is the Grand Poobah of all our loyal SoftLayer groupies. Yes, we have groupies. Kevin manages them very nicely and helps them with just about any crazy request they might come up with. Of course there are no Flintstone pictures of Sam either, but here is one of Sam in the SLintstone era just to fair to everyone! You can check out his handy work on our Facebook fan page!

So what you have witnessed here is a few of the many people that make SoftLayer as cool as we are. We all work hard to make sure of one thing; that our customers are as happy at the SoftLayer server quarry as we are!

If you read through some of the previous blogs on this site such as our CEO’s “SoftLayer Thinks ‘Outside the Box’” or the blog written by one of our super developers, Mr McAloon, entitled “Simplicity”, or Mr Rushe’s “An Interview with an elevator” (OK – that has nothing to do with what I’m referring to, but it’s one of the funniest blogs on this site), one thing you’ll notice is that at SoftLayer, we try to automate and simplify things for the customer. Our customer portal has a LOT of customer features. There are automated OS reloads, the ability to boot into a rescue kernel, the capability to add IP addresses on demand, add and configure a firewall or a local or global load balancer, the ability to edit your DNS settings (forward or reverse) and – my favorite – the ability to reboot your server via IPMI or the power strip. You can also manage your CDN services, monitor your NAS or iSCSI storage, configure backups, use the free KVM services, check your bandwidth and of course, handle all of the usual things like opening support tickets or checking your invoice. Or, if you want to integrate any or all of those features into your own management system, there is a full API available for your use.

With all of that functionality in the portal, one of the challenges we continuously run into is educating new customers on all of the features. Not just educating them on how to use the features – but that the features actually exist in the customer portal. A lot of our customers are either new to On-Demand IT Infrastructure Services (aka the hosting environment) or come from other competitors that only offer a fraction of the features that we are able to provide. For instance, you would be amazed at how many customers open “reboot” tickets. While we respond to tickets quickly, it is actually faster for the customer to click on the “reboot” button in the portal than to click on the “create new ticket” link in the portal and then type out a reboot request.

As ways to address that issue, we created a private customer forum so that customers can share ideas, comments and suggestions with each other. We have also not only created the KnowledgeLayer FAQ database, but we have integrated that directly into the support ticket feature of the portal (when you open a ticket, the FAQ system will automatically recommend related fixes before you even submit the ticket). We also have tutorials directly linked inside the portal and even have all of our API documentation available for review.

So one of the challenges we have at SoftLayer isn’t just creating and deploying the new features and services that keep us out in front of the pack, but educating our customers of their existence and their ease of use. BTW, that’s a great problem to have!

Welcome to the Wide World of SLAS 70 (for those who use Google it is really SAS-70 Type II)! Shh, don’t tell anyone but we are in the final audit stages now (under review). For some reason this is supposed to be a big secret, well not really or I wouldn’t be telling you about it. Maybe it is just so you don’t jinx the whole process.

If you are over 40 and doing the things you are supposed to do at the doctor then you have a pretty good idea of what SLAS 70 is. So we open our company up for the scope. I will just leave it at that. SLAS 70 is us telling the world how good we take care of our processes and procedures when dealing with our customers and also our internal controls. Since we would like to have more publicly traded companies as customers we need to be compliant according to my brother (the other Kinman blogger) and Mike Jones, CFO extraordinaire. It really is a good thing because the investors in these public companies like the fact that they use compliant companies, well they demand it apparently, which helps give them peace of mind in their investment. I’m not sure how many investors are left in the world but even the 1 or 2 still out there count for something. In a nutshell, we have to be good at what we do and we have to let an accounting firm come in and look at everything we do and then sign off that we are the best. WE know we are the best and now we will show the world.

SLAS Hint #1 – PDF! So I don’t know how many sheets of paper I saved with good old fashioned PDF conversion but I can bet I saved at least 2 trees. If I had to print out all this stuff like in the old days it would have been scary.

SLAS Hint #3 – Time! It is all about how much time you have or don’t have. I should have kept track of the hours for this project but they all ran together at some point last week. I thought Thursday was Friday, and was really bummed out when I was told it wasn’t. I bet everyone has done that at least once.

Ok, well it is back to staring at the screen and hunting and fishing, not the fun kind either, just for data needed for the scope! I don’t want my new buddy, the auditor, to miss me since we hang out and email so much. Hey, can someone please pass me some latex gloves?

Early one Friday morning, as I made my way into the NOC for my usual shift, I was greeted by my fellow workers for what would seem a normal workday. Immediately upon my arrival, I was given my first task of the day by David, the overnight datacenter manager – get some coffee ready for the tour we have going this morning. I will submit to you, the reader, that making coffee should be no daunting task, but even the best of are tricked sometimes by simple machinery. With that in mind, I went ahead and made the coffee as I had done many times in the past with similar coffee makers. At that point I figured I was good to go, and went about my business. When I went to double check the coffee, I was treated to a nice little puddle of steaming coffee in the break room. While mopping up my little creation, a few things came to mind:

While I had done this many times in the past with similar coffee pots, this one has some bells and whistles that the old school ones didn’t have. New buttons and some water piped in directly.

I was going to have to attack this problem head on, because there was no way I was going to deprive our customers of their caffeine!

As I thought about it more while bringing the sopping wet trash downstairs, I realized this little debacle wasn’t too far distanced from what I do here at SoftLayer.

Imagine this: You have a piece of hardware you use all the time. It’s a great piece of hardware, it rarely fails, and its principles have remained steadfast over the years. Suddenly, though, a new firmware version is released, and your “tried and true” methods are no longer working! Working in a dynamic business like the web hosting industry, things change at the blink of an eye. Its quite the detriment to get stuck in the rut of “that’s how I’ve done it for years!”. Flexibility and the ability to adapt are crucial. Otherwise you find yourself with a server that doesn’t work, or a pot of coffee overflowing in the break room.

Second, every day offers new challenges from the last. Much like the dreaded coffee maker, there are many a problem that can be solved with a little perseverance (or a user’s manual – something the coffee machine didn’t feel the need to grace me with, unlike the Manual pages, or the trusty F1). Every day, it seems a new hardware problem presents itself. It’s up to the Server Build Engineers here at SoftLayer to ensure that those problems don’t keep the customer from getting their server in the window in which it was promised. Whether it’s incompatible hardware, or just a piece of gear that doesn’t want to play nice, you can rest assured that the SBE’s here will be on their toes solving problems rather consistently.

And to answer David’s question about how can he trust someone to answer tickets when they can’t brew coffee? Actually, easy – I figured it out. Not to mention, I’m sure I saw the lucky people leave the tour today with a little extra pep in their step. I can only hope it was only *partially* because of the coffee.

Do you remember sitting back in a high school class saying to yourself: “I am never going to use this in the real world once I grow up!” Well I often felt that, especially when I was in the Student Computing Services program at Henderson Bay High School in Gig Harbor. The year was 1998 and Henderson Bay had just landed a grant from the Intel Corp, which made us a certified Intel refurbishment program.

Intel and the computer teachers worked on a program, which they donated old hardware (Pentium 90’s when Pentiums 120’s where being released) to our school. We then had to develop a streamline operations program of building these computers, using an imaging system to install the OS and applications. Then we had to create an inventory tracking system to track them prior to giving them away to other schools in Washington State. Over the course of 3 years, I think we deployed a few thousand machines throughout the state, while Tigard High school in Oregon was doing the same thing.

I was working a long day at the beginning of this month for our beloved truck day here at SoftLayer, this is when we get all of the servers the SLales team will sell for the month. All local staff is required to be there and work long hours. We streamline the process so well that within hours we have unboxed, sorted all the parts, double checked the inventory, and deployed the 500 or so servers into the rack each month at each location.

The process reminds me of the truck days we used to have when the Intel truck would show up with cases, motherboards, processors, ram, and hard drives. All which had to be put together. We never had a problem building and deploying boxes for the schools ten years ago, and that is because the teachers ran it as a company. We had a staff of students that operated as inventory control managers, project managers, systems administrators, and front line technical support, and hardware technicians supporting each school district that we donated systems too. The process was overseen by two teachers, that ran the SCS program, and it was there fine detail that kept 50 students running this mock company.

It’s the same fine detail that the operations team at SoftLayer has taught us while doing truck days. I first dreaded my first or second truck day – something about working doubles didn’t appeal to me. After one or two of them, I started to like them. It’s a wonderful way to start off the month. Now if you would have asked me during a high school truck day, if I’d be looking forward to doing it again in ten years, I would have told you that you’re out of your mind. Ask me today and I will tell you it was one of the greatest processes that I learned during my high school years (Along with ditching English to hang out in the computer lab).