Author Archive: Steve Kinman

As a "techy turned marketing turned social media turned compliance turned security turned management" guy, I have had the pleasure of talking to many different customers over the years and have heard horror stories about data loss, data destruction, and data availability. I have also heard great stories about how to protect data and the differing ways to approach data protection.

On a daily basis, I deal with NIST 800-53 rev.4, PCI, HIPAA, CSA, FFIEC, and SOC controls among many others. I also deal with specific customer security worksheets that ask for information about how we (SoftLayer) protect their data in the cloud.

My first response is always, WE DON’T!

The looks I’ve seen on faces in reaction to that response over the years have been priceless. Not just from customers but from auditors’ faces as well.

They ask how we back up customer data. We don’t.

They ask how we make it redundant. We don’t.

They ask how we make it available 99.99 percent of the time. We don’t.

I have to explain to them that SoftLayer is simply infrastructure as a service (IaaS), and we stop there. All other data planning should be done by the customer. OK, you busted me, we do offer managed services as an additional option. We help the customer using that service to configure and protect their data.

We hear from people about Personal Health Information (PHI), credit card data, government data, banking data, insurance data, proprietary information related to code and data structure, and APIs that should be protected with their lives, etc. What is the one running theme? It’s data. And data is data folks, plain and simple!

Photographers want to protect their pictures, chefs want to protect their recipes, grandparents want to protect the pictures of their grandkids, and the Dallas Cowboys want to protect their playbook (not that it is exciting or anything). Data is data, and it should be protected.

So how do you go about doing that? That's where PLEB, the weird acronym in the title of this post, comes in!

PLEB stands for Physical, Logical, Encryption, Backups.

If you take those four topics into consideration when dealing with any type of data, you can limit the risk associated with data loss, destruction, and availability. Let’s look at the details of the four topics:

Physical Security—In a cloud model it is on the shoulders of the cloud service provider (CSP) to meet strict requirements of a regulated workload. Your CSP should have robust physical controls in place. They should be SOC2 audited, and you should request the SOC2 report showing little or no exceptions. Think cameras, guards, key card access, bio access, glass alarms, motion detectors, etc. Some, if not all, of these should make your list of must-haves.

Logical Access—This is likely a shared control family when dealing with cloud. If the CSP has a portal that can make changes to your systems and the portal has a permissions engine allowing you to add users, then that portion of logical access is a shared control. First, the CSP should protect its portal permission system, while the customer should protect admin access to the portal by creating new privileged users who can make changes to systems. Second, and just as important, when provisioning you must remove the initial credentials setup and add new, private credentials and restrict access accordingly. Note, that it’s strictly a customer control.

Encryption—There are many ways to achieve encryption, both at rest and in transit. For data at rest you can use full disk encryption, virtual disk encryption, file or folder encryption, and/or volume encryption. This is required for many regulated workloads and is a great idea for any type of data with personal value. For public data in transit, you should consider SSL or TLS, depending on your needs. For backend connectivity from your place of business, office, or home into your cloud infrastructure, you should consider a secure VPN tunnel for encryption.

Backups—I can’t stress enough that backups are not just the right thing to do, they are essential, especially when using IaaS. You want a copy at the CSP you can use if you need to restore quickly. But, you want another copy in a different location upon the chance of a disaster that WILL be out of your control.

So take the PLEB and mitigate risk related to data loss, data destruction, and data availability. Trust me—you will be glad you did.

On April 7th, the OpenSSL Project released an update to address a serious security flaw (CVE-2014-0160), which allows remote attackers to obtain sensitive information from process memory via crafted packets that trigger a buffer over-read, as demonstrated by reading private keys, related to d1_both.c and t1_lib.c, aka the Heartbleed bug.

SoftLayer Infrastructure

After notification of this vulnerability we began a close examination of our services to determine any that may have been affected. Both the SoftLayer customer portal and API are serviced behind hardware load balancers and neither the hardware load balancers nor the software running on the servers behind them were found to be running vulnerable versions of OpenSSL. This was confirmed by the hardware vendor and direct testing as well. During these tests it was discovered that certain nodes of our Object Storage cluster were running a vulnerable version of OpenSSL. The software was immediately patched to remediate the issue. Although there is no indication that this vulnerability was exploited, the subset of customers potentially affected has been advised of precautionary measures to ensure continued security.

Additionally, our team forced updates to all of our internal operating system update mirrors as soon as patched versions were released by their publishers. Our system automatically checks for and updates all operating system versions hosted on our mirrors, but due to the urgency of this exploit, manual updates were run as quickly as possible to have patched versions available sooner.

SoftLayer Customers

Due to the nature, surface area, and severity of this vulnerability, we recommend revoking all possibly compromised keys and reissuing new certificates for any service secured using the OpenSSL library. The rekeying process can vary depending on your Certificate Authority (CA) and you should contact them if you have questions on how to complete this process. This OpenSSL vulnerability has major security implications for a wide range of operating systems and applications and may necessitate rebooting your hardware (or restarting services) to ensure all services linking against the affected code use the updated version of the OpenSSL library. We also recommend that you patch all of your servers and change passwords as soon as possible. Take this opportunity to review your overall password strategy including password strength and password sharing across sites.

At my house, we share a single iTunes account because as much as I hate to admit it ... I listen to the same music as my 11-year-old on occasion, so why buy the same music twice? I have my iPhone setup to automatically sync via any wireless connection, so I occasionally get new apps when someone else in the house downloads something.

Last week, my 8-year-old handed me his iPod and said, "Dad, can you enter the password so I can install BloodnGuns?" No way. He went through three or four reasons that he thought he needed the game, and I just went about my business. A couple of minutes later, he hands me the iPod again and says, "Dad, can you enter the password so I can install Temple Run?" Being a much tamer game, I said I would, but (knowing my son) I followed that up by saying, "Just remember: Anything you install goes to my iPhone, too." If I entered the password for him for Temple Run, he would be authenticated and could then get BloodnGuns, so I just wanted to remind him that I was born at night, not last night.

The sneaky little guy looked up to me and grinned, "Oh yea, 'cuz of that cloudamajigger thing."

Once I finished laughing, I asked him what he meant by Cloudamajigger, and before he could answer, I told him to wait ... I wanted to document how he would describe "The Cloud." With two other kids at home, I thought it might be an interesting focus group of the way kids are learning about technology, so I made it a family project.

I asked each of them three questions and told them to email their answers to me"

What is "The Cloud?"

Where does "The Cloud" live?

What is SoftLayer?

Here are the responses:

The 6-year-old

The cloud shoots out a ball and the cloud is awesome!

In the sky. It is made out of water.

Where dad works, I think he makes monitors.

The 8-year-old

It's a cloud in the sky and they shot a satellite in it. And they could see all the things you need to see on the internet.

See number 1 (Yes, he really typed that).

Where dad works, he works to make the Internet, and the Internet makes him work.

The 11-year-old

It is a group of people where when you post something everyone will be able to see it.

I don't know.

A company.

You can see that the 11-year-old is darn close to those wonderful teenage years with that loquacious participation ... Wish me luck!

I ask these same questions of people at conferences I attend and get generally the same answers as above. We can write reams of descriptions of the cloud, but in my world, it's simply "The Cloudamajigger Thing."

I was unbelievably busy last week, and surprisingly, the busyness I'm referencing did not even involve my official responsibilities in compliance. I was planning on writing a blog to share some of the fun/insane/ridiculous things that happened, and I thought of a way to mix it up a little and make a challenge out of it for our readers.

Have you ever seen those image-based logic puzzles where you're given a series of images and challenged to put them in order to create a story? Here's an example:

What story are those pictures trying to tell? A boy [6] grabs a fishing pole [4], and finds a fishing hole [5]. He baits his hook [3] and waits for the catfish to quit posing [2] and bite the hook! He takes his catch home, and his mom fries it up [1]. MMMM Good [7]!

You could probably interpret it a different way and "choose your own adventure" where the anthropomorphized fish deep fried the boy ... Depends on how far outside the box you think. The answer the question was meant to have is the one above. Now that you see how it works, I have a logic puzzle for you to try and figure out about what happened during my week last week.

All ten of the pictures below were taken in the span of 56 hours ... If you can come up with the correct story, I'll send you a prize (detailed below). If you can come up with a creative story that isn't correct, I can probably find something to send you as well. Without further ado, here are the pieces of the story [Click for Larger Version]:

If you've been to the SoftLayer Blog this week, you know that we have a "Kids Meal" kind of special going right now where for the next few months if you buy a server and email us, you can get an official SoftLayer Bobblehead! To piggyback on that giveaway, the first person who posts a comment with the correct order of the photos to answer the puzzle (or the funniest answer if no correct answers are posted), will get my personal FULL SET of official SLobbleheads. Yes, the full set! You won't have to wait to place your server orders in the next month to complete your bobblehead collection (though I hope you still keep ordering servers).

The government (FISMA), banks (PCI) and the healthcare industry are huge proponents of two-factor authentication, a security measure that requires two different kinds of evidence that you are who you say you are ... or that you should have access to what you're trying to access. In many cases, it involves using a combination of a physical device and a secure password, so those huge industries were early adopters of the practice. In our definition, two-factor authentication is providing "something you know, and something you have." When you're talking about national security, money or people's lives, you don't want someone with "password" as their password to unwittingly share his or her access to reams valuable information.

What is there not to like about two-factor identification?

That question is one of the biggest issues I've run into as we continue pursuing compliance and best practices in security ... We can turn on two-factor authentication everywhere – the portal, the vpn, the PoPs, internal servers, desktops, wireless devices – and make the entire SoftLayer IS team hate us, or we can tell all the admins, auditors and security chiefs of the world to harden their infrastructure without it.

Regardless of which direction we go, someone isn't going to like me when this decision is made.

There are definite pros and cons of implementing and requiring two-factor authentication everywhere, so I started a running list that I've copied below. At the end of this post, I'd love for you to weigh in with your thoughts on this subject. Any ideas and perspective you can provide as a customer will help us make informed decisions as we move forward.

Pros

It's secure. Really secure.

It is a great deterrent. Why even try to hack an account when you know a secondary token is going to be needed (and only good for a few seconds)?

It can keep you or your company from being in the news for all the wrong reasons!

Cons

It's slow and cumbersome ... Let's do some math, 700 employees, 6 logins per day on average means 4200 logins per day. Assume 4 seconds per two-factor login, and you're looking at 16,800 extra seconds (4.66 hours) a day shifted from productivity to simply logging into your systems.

Users have to "have" their "something you have" all the time ... Whether that's an iPhone, a keyfob or a credit card-sized token card.

RSA SecureID was HACKED! I know of at least one financial firm that had to turn off two-factor authentication after this came up.

People don't like the extra typing.

System Administrators hate the overhead on their systems and the extra points of failure.

As you can start to see, the volume of cons out weigh out the pros, but the comparison isn't necessarily quantitative. If one point is qualitatively more significant than two hundred contrasting points, which do you pay attention to? If you say "the significant point," then the question becomes how we quantify the qualitativeness ... if that makes any sense.

I had been a long-time hater of two-factor authentication because of my history as a Windows sysadmin, but as I've progressed in my career, I hate to admit that I became a solid member of Team Two-Factor and support its merits. I think the qualitative significance of the pros out weigh the quantitative advantage the cons have, so as much as it hurts, I now get to try to sway our senior systems managers to the dark side as well.

If you support my push for further two-factor authentication implementation, wish me luck ('cause I will need it). If you're on Team Anti-Two-Factor, let me know what they key points are when you've decided against it.

On Tuesday, Summer posted "Giving: Better Than Receiving," a blog about all of the organizations SoftLayer has supported in 2011, and I'm one of the lucky SLayers on the new Charity Committee. We recently began this initiative to oversee charitable donations at SoftLayer and (more importantly) to encourage all employees to step-up and make a DIFFERENCE. Whether by volunteering or financially supporting a local charity, the idea is that we all participate in our community and try and help in some way.

One of the best examples of an organization that does amazing things for communities and people who deserve a little extra love is the TV show, "Extreme Makeover: Home Edition." I've always loved the show, and I'm only quasi-embarrassed that I've shed a tear or two when the crowd shouts, "Move that bus!" and the homeowners see their brand new home. If you aren't familiar with the show, the EM:HE team finds deserving families who, for one reason or another, need a new home, and over the course of one week, the EM:HE crew and a slew of local volunteers set to work to rebuild or remodel the home.

You can imagine the amount of supplies, coordination and man-hours that go into building a new home or completely remodeling it in just one week. That's where the community and local businesses get involved: Supplies are donated by companies, and the work force is made up of show employees, people from the sponsoring companies, and an average of 2,500 volunteers every episode.

With that generous involvement, the challenge becomes coordinating the massive amount of work, people and projects to get everything done in a short period of time. That's where the Internet comes in. How can the show maintain an online presence for vendors, sponsors and fans of the show? Each of them plays an important part in the show's success, so they need to be kept "in the know" with the most up-to-date information. And that's where we come in.

This philanthropic show definitely meets the requirements of SoftLayer's Charity Committee, and when the show was nominated as a prospective organization to support, we immediately set plans in motion to figure out how we could help support the show and the deserving families getting new homes.

We've donated $25,000 in free hosting services this season to support the show's online presence. We'll be providing a place for vendors who donate to gain some visibility and a place for fans to watch videos and keep up with the show ... And that's no small task: The site receives about 6.8 million monthly impressions.

As Summer mentioned in her post, this is just one of the many ways we're reaching out to support organizations that are doing great work. Let us know what charities matter the most to you, and we'll get them on our radar. We're always looking for ways to get involved, and the first step is learning about who's doing this kind of amazing work for such a great cause.

As far back as I can remember, I hated homework. Homework was cutting into MY time as a kid, then teenager, then young adult ... and since I am still a "young adult," that's where I have to stop my list. One of the unfortunate realizations that I've come to in my "young adult" life is that homework can be a good thing. I know that sounds crazy, so I've come prepared with a couple of examples:

The Growing Small Business Example
You run a small Internet business, and you've been slowly growing over the years until suddenly you get your product/service mix just right and a wave of customers are beating down the door ... or in your case, they're beating down your website. The excitement of the surge in business is quickly replaced by panic, and you find yourself searching for cheap web servers that can be provisioned quickly. You find one that looks legit and you buy a dozen new dedicated servers and some cloud storage.

You alert your customers of the maintenance window and spend the weekend migrating and your now-valuable site to the new infrastructure. On Monday, you get the new site tuned and ready, and you hit the "go" button. Your customers are back, flocking to the site again, and all is golden. As the site gains more traffic over the next couple of weeks, you start to see some network lag and some interesting issues with hardware. You see a thread or two in the social media world about your new shiny site becoming slow and cumbersome, and you look at the network graphs where you notice there are some capacity issues with your provider.

Frustrated, you do a little "homework," and you find out that the cheap service provider you chose has a sketchy history and many complaints about the quality of their network. As a result, you go on a new search for a hosting provider with good reviews, and you have to hang another maintenance sign while you do all the hard work behind the scenes once again. Not doing your homework before making the switch in this case probably cost you a good amount of sleep, some valuable business, and the quality of service you wanted to provide your customers.

The Compliance-Focused Example
I still live, eat, and breathe compliance for SoftLayer, and we had an eye-opening experience when sorting through the many compliance differences. As you probably recall (Skinson 1634AR15), I feel like everyone should agree to an all-inclusive compliance model and stick to just that one, but that feeling hasn't caught on anywhere outside of our office.

In 2011, SoftLayer ramped up some of our compliance efforts and started planning for 2012. With all the differences in how compliance processes for things like FISMA, HIPAA, PCI Level 1 - 4, SSAE16, SOC 1 and SOC2 are measured, it was tough to work on one without affecting another. We were working with a few different vendors, if we flipped "Switch A," Auditor #1 was happy. When we told Auditor #2 that we flipped "Switch A," they hated it so much they almost started crying. It started to become the good ol' "our way is not just the better way, it's the only way" scenario.

So what did we do? Homework! We spent the last six months looking at all the compliances and mapping them against each other. Surprisingly enough, we started noticing a lot of similarities. From there, we started interviewing auditing and compliance firms and finally found one that was ahead of us in the similarity game and already had a matrix of similarities and best practices that affect most (if not all) of the compliances we wanted to focus on.

Not only did a little homework save us a ton of cash in the long run, it saved the small trees and bushes under the offices of our compliance department from the bodies that would inevitably crash down on them when we all scampered away from the chaos and confusion seemingly inherent in pursuing multiple difference compliances at the same time.

The moral of the story: Kiddos, do your homework. It really is good for something, we promise.

As many of you know SoftLayer is going global. Our Singapore DC goes live TOMORROW, and Amsterdam will follow suit shortly, so we put together a little "jingle" that I think you might know. It might be September, but if the stores are already putting out holiday items, Christmas songs should be fair game in October ... And since we are entering that last stretch of work before those great end-of-the-year national holidays that give us a few days off, we can use a classic tune to help us power through.

To those of you who love the song, "Santa Claus is Coming to Town," you may not want to play the video below. To those who want to rubberneck at our goofiness and join us in a little fun ... play away:

If you want to sing along at home (because who wouldn't?), here are the lyrics for your karaoke pleasure:

Shout-outs go to all the SLayers who indulged us in this little song. We hope it's less embarrassing than you expected ... And if it's more embarrassing, we hope it's as terrible and catchy as "Friday."

Tip: If the song is stuck in your head now, one great way to distract yourself from it is to go and order a server in Singapore!

Once again the Dallas Cowboys let a game they weren't supposed to win slip away from them in the 4th quarter. Again it was Tony "oops" Romo that had a hand (or "didn't have hands") in the loss. I can't blame it all on him as I saw many problems that led up to the defeat. I, as a master football coach of 4-6 year-old flag football, could write multiple paragraphs on that subject, but because this is a social media blog, I will get back on topic.

After last night's "4th quarter of doom" that probably led to crazy nightmares for my sleeping kids (I may have been yelling loudly and often), I decided to open Twitter to see what everyone in the world thought about the game. I have to admit I was a little shocked at how many Cowboy haters are out in the wild. Of course the game was trending, and the conversation was ... diverse: You had your die-hard Cowboy fans that were saying, "Shake it off, you weren't supposed to win anyway." You had your fair weather fans that were saying, "Great, another season opener loss, I guess I'll follow the Texans instead." You had the fans of other teams that were saying, "Haha, the Cowboys lost again – Go (Insert your team here)!" And, of course you had the pure Cowboy haters who were saying, "#$%^#$%^#$ the Cowboys they #$%#$% and #$%# and then #$%#$%. Eat it!" I would say most were Cowboy haters, and most of the tweets were not even close to being rated PG-13.

Stay with me now ... I'm finally onto the real topic.

Social Media
What I saw on Twitter last night was real Social Media to me. It was current, real time, opinionated, cool and sad all at the same time. It encapsulated the thoughts and reactions of the public to something that was happening or just happened. Why is social media cool? A couple of weeks ago when the earthquake struck the northeast, people were saying that they received tweet updates of the ground shaking and notifications that an earthquake hit seconds before they felt the tremors in their area. Think about that and how many possible uses that has in lots of different industries. X happens, Y needs to know about it right away, Z tweets it or posts it on Facebook (or any of the 2000 other social apps out there), and like magic you have the information almost before you are supposed to. That's viral social media.

Social Marketing
Social Marketing isn't nearly as sexy. It's only and exactly what it sounds like. We do it at SoftLayer: You see tweets from us talking about press releases, new products, our new website, our new international locations and some of the other value we provide to customers because we know how easy it is to miss some of the best stuff in the noisy social sphere. It helps us build our brand and helps with awareness by getting our name in front of people who may not have seen it otherwise. It drives traffic to our website and straight to our order form. It is significant to our bottom line.

The challenge with this kind of engagement is that the volume of content can seem overwhelming to some. Some customers only want to hear the viral social media kind of stuff with up to the minute news (which is our vision for @SoftLayerNotify), but it's tough to abandon the social marketing piece because it's been so measurably successful for us.

With that being said, we want to hear from you about what you like and don't like about our social engagement. What you would like to see more of? What would you like to see less of? Do you like it? Do you hate it? We're definitely listening ... Well as long as we're not busy getting ready for the next flash mob.

Dallas -- In a world where auditor to auditor reports are out of control and we have a mountain of complex compliances to worry about, one competent compliancy controlled certification of compliance finally comes forth (and not a minute too soon).

"This new groundbreaking idea will change the lives of many competing auditing firms, law firms, accounting firms and so on," says Steve Kinman. "I spend countless hours reading controls for one report and different controls for another report, and the only difference is the verbiage and format."

The new Skinson 1634AR15 Certification combines your SAS70, SSAE16, ROC, VOC, SOC, NIST, SARBOX, PCI, OMB, ACART, CFDA, HIPAA and SAFE HARBOR compliance into a single report using a set framework that automorphs based upon which auditor is touching the report or viewing it in the state of the art Skinson Portal.

"The Skinson portal is mind-blowing," says Val Stinson. "The automorph feature is something straight out of the movies. It knows who is reading and can change the wording on the fly. This keeps auditors from scratching their heads when the words in the report don't match the words their instruction book."

The introductory price for full Skinson 1634AR15 Compliance Certification is $1,000,000 USD. This is all-inclusive and will sufficiently cover all of your compliance needs.

About Skinson
Headquartered in Dallas, Texas, Skinson is a fictional company that likes to poke fun at the difficult job of compliance in the world. While we find that it can be overwhelming at times, we understand that compliance is a necessary evil. We would like to note that something like we dream about above would be very nice and would save the world a ton of work and cut down on our carbon footprint considerably. If you are in a position of control and can make the above happen please help us!!

On a side note, SoftLayer will do everything we can to help you with any compliance you need. Just ask your local sales team for help, and they will find the right person and get you in contact.