Posts Tagged 'Management'

If you read my previous blog, “What Does it Cost (Part 1) - The Overview,” you may be interested to delve deeper into the conversation and math behind how all of this adds up. Essentially asking yourself “is it better to build infrastructure yourself?” is a good thing and you will inevitably try to ask yourself what does it cost to do so versus looking into “what would it cost to have SoftLayer do this for me since this is what their core competencies reside in?”.

Remember that one of the big lessons we can learn and that I re-learned at the conferences I attended is that your people are your biggest assets. This lesson is showcased and repeated several times and for good purpose, since this seems to be a time tested rule. While your people are a biggest assets they can also easily be one of your biggest costs especially if they are not managed properly. Every business should have a growth model but one thing that can hold you back is the cost of growth (or your growing pains).

Think about the amount of people you need when you run everything inside and what that will wind up costing. If your business, network, and uptime are all mission critical you’ll also need to take into consideration the number of people needed to make sure a facility is 24*7. You will need someone to fix a drive that brakes and needs to be replaced at 3:42AM, won’t you? Take the number of people that you think you’ll need and now consider what would happen if you were to double in size in a single year (or you could use your own timeline in your head). Would you need double the people or possibly more when you consider the needs of managers to make sure everything was in line with your business strategy? What would the cost be that you would need to pay when considering more than just their salaries.

Think of the other things that do not jump out at you immediately like taxes, insurance, a 401K plan, office space, other liabilities, etc. Gary Kinman (VP of Accounting and Finance) estimates that the cost of each additional employee is about 15-20% more than just the cost of their salary without including things like office space. This is one of the biggest aspects often overlooked, because it not only takes new people you would need to hire, but how it can monopolize time and production you would get otherwise from people you already have on staff.

Now, if you remember from part one I mentioned how Opportunity Costs are some of the biggest costs in the differences between how SoftLayer can help you versus doing things yourself. If you reverse the previous scenario and say that after you’ve just doubled in size there is a bust in the economy which causes you to have to contract. For starters the easiest way to cut back on spending is in people, so you may have to lay people off and ultimately make you the bad guy. Now here is where ugly gets really gruesome.

If you talked yourself into how cheap it can be to buy and do everything yourself you are in a real tight spot because now you may not have the necessary people to run all of your infrastructure, or in an even worse case scenario you may not even need it. What this spells out is that you keep something that cannot be used even though you are paying for it, and you had to let people go just to keep the rest of the boat afloat. Didn’t we say that our people are our most important asset earlier? You can’t always know what kind of worker someone will be when you hire them or how things will work out, but you do want to put yourself in a position to keep the good ones that you trust to push your business forward around and happy.

All right, that is enough doom and gloom scenario. Let’s look at this subject from another angle. As you grow in size generally everything you have and everything you use will grow right along the company. We covered the fact that it will probably become more and more obvious that you’ll need more people to do the work for your business. Hiring systems administrators, DBAs, and development staff can all be good moves that would impact your business specifically; however, are you putting them in the best position for them to be successful? Have you ever seen that show “Undercover Boss”? It seems that in a lot of the episodes you would see that a CEO was not cut out for doing a lot of other jobs in the company and would have a much greater appreciation of everyone who did all of those jobs and how hard they work. Sometimes they would have comments about if they were really trying to get that job they wouldn’t last long. Keep that thought in mind when asking these same Sys-admins, DBAs, and development staff to do jobs that they do not specialize in.

Taking your people in positions where they may get a grade of an “A” or a “B+” and putting them into different positions where they may get a “C-“, “D”, or even an “F” will not likely be good for production levels, decrease levels of morale, and will also likely tank the investment value made in the employees themselves and/or the infrastructure you purchase.

Bottom line is that the way the world is evolving is to work smarter, lessen risk, and (in drawing back to part 1) get more out of having less. The best way to avoid unnecessary risk is to not overextend yourself in the first place, and to stay in a position of flexibility so that you can react and adapt to the market around you. This is what SoftLayer is built for; keeping you with the most options in order to increase your ability to innovate and execute without sacrificing any level of control and without costing large sums of upfront capitol.

I am guessing that about 9 times out of 10 if you take the time to sit down and do the math it all makes perfect sense.

SoftLayer was nominated for the Best Places to Work in DFW this year. I have no doubt that SoftLayer will win this!

The commitment and loyalty Management has shown to SoftLayer has reflected to the employees. There is a saying “Treat your employees just like customers or even better.” SoftLayer understands that and therefore one of the biggest assets of the company is the employees. The company offers employee break rooms with refrigerators, microwaves, and a various assortments of drinks and snacks. The satisfaction and motivation encourages employees to do a better job which translates into customers being more satisfied as well.

SoftLayer also honors employees who have done an outstanding job for that year and offers a few fun awards for their hard work.

Since we are growing and doing very well in the industry we feel that supporting the community is a key initiative. We recently donated to the Haiti Relief Fund then engaged the employees to get involved also by offering an additional company match.

As previously noted in a few blogs we are also a firm believer of being “green” and not only in the Data Centers. A recycling bin is put in each break room and the employees are encouraged to use them. Employees are also encouraged to print as little as possible to save paper and if they must print they must recycle the paper rather than throwing it away. Except for that really secure SAS 70 affected top secret stuff that must be shredded.

SoftLayer is growing at a rapid pace and who would not want to work for a company that is growing? I feel privileged to work at SoftLayer and embrace the “challenging but not overwhelming” mantra.

How many times have you received a “task” through email with no priority or due date attached? Just “Hey, do this…” with nothing more. It leaves you wondering when this particular “task” is supposed to be completed or how important this task may be. What if you’re slammed with about 5 different items at once and the email with the “task” disappears into the mass of emails you receive all day? Now you have the author of this “task” upset because their task was not completed by the time they didn’t specify in the email lost in your inbox. It’s a disaster just begging to happen.

Emails get lost. Task notes get thrown away by the cleaning crew. The dog ate my task. In using a task management system, none of these situations could ever happen.

A Task Management System is either a frightening or salvatory three words for the disorganized among us. It’s a savior for those desiring efficiency and a nightmare for those unwilling to change.

Wow, you are really convincing! So, what type of task management systems are out there? I’m glad you asked that question...

Task Management Systems range from the simplest (Ta-da Lists - http://tadalist.com/) to the more advanced (JIRA - http://www.atlassian.com/software/jira/). Both, of which, could meet your needs exceptionally well.Wow, JIRA looks really awesome! What are some pros and cons of the task management system? Another excellent question… PROS:

Task organization

Task prioritizing

Task collaboration between employees

Task status updates

Custom reports for Tasks

Task history CONS:

New system to learn.

That’s really about it, honestly.

It’s really a no brainer that the task management system is a major improvement over basic email and can bring about high efficiency in the work place.

Those who keep a close eye on the menu options in the customer management portal will have noticed that recently there was added an option under Security where you can now order SSL certificates. For those not familiar with SSL, a certificate is used by an application to establish identity and provide encryption services. Naturally you do not have to order your SSL certificates through us. Certificates ordered other places will work just fine on your server here. Certificates ordered here will work fine elsewhere.

So why order your SSL through SoftLayer? To me, its a convenience and security thing. Ordering with us is convenient because you can place and manage the order via the portal just like you manage aspects of your account already. Management includes being able to see when your certificates are going to expire and the ability to renew them. If the certificate file itself is deleted by accident you can get a copy of it e-mailed via the portal. From a security point of view you already have a billing arrangement with us so why give your credit card information to another party?

I can see someone thinking "But is that safe.. what if I leave SoftLayer?" Yes, it is safe. The only information you have to provide to us in doing the ordering is the Certificate Signing Request and some billing verification. Both of these are things that would be provided to any SSL vendor. The private key, which is the core of SSL security, is not kept or handled by SoftLayer. The private key is generated and remains with your administration staff on your server.

Let us chat about the private key for a moment. The private key is meant to be known only by the server applications to which it is assigned on your server. If it is lost, corrupted, deleted, whatever it will require a new certificate. What this all means is that you should only allow people you really trust access to the private key and above all you must keep a good, safe backup of the file. SoftLayer support can perform quite a bit of server voodoo but recreating a lost private key isn't an option.

I'd invite anyone with a bit of time to experiment with the SSL functionality we offer. You might find something useful for your business.

I remember when I first started here at SoftLayer it was quite exciting and nerve racking at the same time. You see I came from the telecom industry, and I worked for a huge company that had 100,000+ employees. Basically, I did the same thing everyday. I learned a lot when I first joined this huge company, but I felt like my career had become stagnant and I needed a change. I decided to look for a job at a small company and be challenged everyday. Man, did I find the perfect job!!! Anyway, back to the point. At first, I didn't realize how advanced SoftLayer was till I began to look more into the company and the industry we are in. The more I dug the more I was impressed and excited to be a part of something revolutionary.

I know we all have seen the movie "Charlie and the Chocolate Factory" (old school version) or at least most of us. I know it sounds cheesy but that is the best way to describe how I felt when I first joined. I felt like Charlie. I was just amazed to see what I saw inside the workings of SoftLayer. I had no idea that some of the tools/services/automation SoftLayer had done was even possible. The best way for me to describe the management of SoftLayer is they are the Willie Wonka's of our industry. Some people may think SoftLayer's ideas are radical or even impossible, but we don't. All the great people here at SoftLayer work together to make the impossible possible.

While other companies try to mimic us, we are busy turning fantasies into realities.

Recently I had the chance to attend the annual Beyond Budgeting Round Table (BBRT) conference to help me keep up on my CPE credits. Those darn accounting licenses have to be maintained, ya know.

I was pleasantly surprised at the conference that SoftLayer was already doing the crux of what this group preaches – namely, that assembling an annual budget and trying to live by it is a colossal waste of time!

One speaker pointed out that budgeting originated back in medieval times long before the Industrial Revolution. During those days, the feudal system was the order of the day. Landowners allowed people to live on their land and raise crops. Once per year, when the harvest came in, the landowners received payment from the people living on the land in the form of a share of the crops or a share of the gold for which the crops were sold. Since the landowners were paid once per year, they had to plan how to make their annual payday last for a whole year. You guessed it – this plan was called “the budget.”

Unfortunately, most companies and organizations today use this horribly outdated financial management technique to run their business in the fast-paced information age economy of today. In most cases, this just flat doesn’t work.

For example, one of the speakers was the CFO of a very large healthcare organization. He said that back in the days when they produced an annual budget, there were 240 budget managers that spent 90 days of full-time effort to produce the annual budget. That equates to 60 man-labor years of total time to produce that budget. If you assume that each of those managers averages $50K per year in compensation, the cost of producing that budget is $3 million. What’s worse is that the CFO said it was worthless before the final version was printed because it was built on stale fundamental assumptions that were several months old.

Once these obsolete documents are produced, they become static financial contracts. They limit spending for each department, and this isn’t always a good thing. Some departments may see some fantastic market opportunities develop halfway through the year, but they can do nothing to take advantage of them because they would exceed their budget. On the other hand, some departments can be allotted too much money, so they go on wasteful spending sprees at year end to be sure and use up their budget or else lose that funding next year. People often ask for permission to exceed budget, but usually no one gives back any unused budget dollars. Even worse, management compensation is often tied to these obsolete financial contracts. Business schools are awash with case studies of bad business decisions that were made to maximize bonus compensation in relation to the budget.

From the beginning, SoftLayer realized the futility of producing an annual budget. In the rapidly developing business of web hosting, the landscape can dramatically change much more quickly than an annual cycle. So we implemented the policy of maintaining a rolling forecast that is updated to the best of our current knowledge each and every month. This practice has served us well, and is one of the “best practices” adopted by the BBRT.

Another best practice recommended by BBRT is to maintain multiple forecast scenarios that factor in macroeconomic possibilities. Then as reality develops, you have a better handle on the tactics to implement because you now know what most of these decisions should be in advance. At SL, we will be implementing the multiple scenario practice over this summer.

When it comes to managing a server remember you can never be to careful. In this day and age we face a lot of things that can damage and even take a server to its knees here’s a few things for everyone to consider.

Anti-virus:

This is a must on systems open to the net now days. There are always nasty little things floating around looking to take your server apart from the OS out. For windows servers there are a multitude of choices and I’ll just mention a few that can help protect your goods. You can use several programs such as avast (which offers a free edition), ClamWin (open source), Kaspersky , and Panda just to name a few. I would suggest before installing any of these you check links such as http://en.wikipedia.org/wiki/List_of_antivirus_software to name one that provides a list of several choices and their compatibility. You may also want to read reviews that compare the available options and give you an idea of what to expect from them. This will allow you to make an informed choice on which one works best for you. Now with linux there are also several options for this including the well known clamav which from personal experience works really well and can be installed on a variety of linux disro’s(aka distributions). It’s very simple to use and may prevent you from headache later on down the road.

Firewalls:

Firewalls are a double edged sword but are most defiantly needed. When it comes to firewalls you can protect yourself from quite a bit of headache however if setup to strict you can block positive traffic and even yourself from reaching your server but in the long run a defiant way to help protect your server from unwanted visitors. A lot of firewalls also have modules and add-ons that further assist in protecting you and securing your server. If in doubt it’s always a good idea to have a security company do an audit and even a security hardening session with your server to make sure you are protected the best way possible.

Passwords:

This is probably one of the most important this you can do to secure your server. Use strong passwords (no using password or jello is not a secure password even if it is in all caps) and if you are worried about not being able to come up with a secure one there are several password generators on the web that can come up with secure ones to assist. Passwords should contain caps letters, numbers, symbols, and should be at minimum 8 – 10 characters (the more the better). It’s the easy to remember and easy read passwords that get you into trouble.

Armed with this information and so much more on security that can be located on the web using the great and all powerful Google should be a good start to making sure you don’t have to worry about data loss and system hacks. Also remember no matter how secure you think you are make regular backups of all your important data as if you server could crash at any time.

One thing I have noticed about my SoftLayer family is the number of musicians here. I spent nine years as a musician, working the bar/festival circuits all over the midwest. When I arrived at SoftLayer, people jokingly asked if I was joining the SoftLayer band due to my previous experience. Just looking at the Operations Management Team, most of us have written/performed and many continue to. The more I thought about it, I think that this is a good thing.

There have been a number of scientific research projects about the academic performance of children and teens who are involved in a music program at school. The mixed left/right brain activity of music lends itself to problem solving and critical thinking as well as creativity which are required by many jobs. A musician's mind is capable of working complex geometric patterns into physical movements in coordination with muscle memory. Once more advanced levels are reached, a musician is capable of not only composing music, but also improvisation. These aspects are similar to many necessary thought processes used in the office.

I'd like to think that these mental processes allow me to think better on my feet, deal with change, and have a global view of the projects that I am involved with. Maybe that's why our Inventory, Hardware, & 2/5 Datacenter Managers are musicians. Not only that, but our Director of Operations is an avid musician. If you count the actual system admins who are musical, then you would be adding another 3 people.

So if all the research is correct, this may have something to do with the quality of operations here at SoftLayer. I wonder if being a musician had anything to do with my interview and its result. I know that it would catch my attention knowing that an applicant had experience with music or any other analytical/creative endeavor, especially if it has been shown to improve overall performance or intelligence.

Maybe I should write a SoftLayer song - an anthem to our Datacenter or a love song about the management network and IPMI?

Recently on one of our technical forums I contributed to a discussion about the Windows operating system. One of our director’s saw the post and thought it might be of interest to readers of the InnerLayer as well. The post focused on the pros and cons of Windows 2008 from the viewpoint of a systems / driver engineer (aka me). If you have no technical background, or interest in Microsoft operating system offerings, what follows probably will not be of interest to you—just the same, here is my two cents.

Microsoft is no different than any other developer when it comes to writing software--they get better with each iteration. There is not a person out there who would argue that the world of home computers would have been better off if none of us ever progressed beyond MS-DOS 1.0. Not to say there is anything wrong with MS-DOS. I love it. And still use it occasionally doing embedded work. But my point is that while there have certainly been some false starts along the way (can you say BOB), Microsoft's operating systems generally get better with each release.

So why not go out and update everything the day the latest and greatest OS hits the shelves? Because as most of you know, there are bugs that have to get worked out. To add to that, the more complex the OS gets, the more bugs there are and the more time it takes to shake those bugs out. Windows Server 2008 is no different. In my experience there are still a number of troublesome issues with W2K8 that need to be addressed. Just to name a few:

UAC (user access control) - these are the security features that give us so much headache. I'm not saying we don't need the added security. I'm just saying this is a new arena for MS and they still have a lot to learn. After clicking YES, I REALLY REALLY REALLY WANT TO INSTALL SAID APPLICATION for the 40th time in a day, most administrators will opt to disable UAC, thereby thwarting the added security benefits entirely. If I were running this team at MS I'd require all my developers to take a good hard look at LINUX.

UMD (user mode drivers) - the idea of running a device driver, or a portion of a device driver, in the restricted and therefore safe user memory of the kernel is a great idea in terms of improving OS reliability. I've seen numbers suggesting that as many as 90% of hard OS failures are caused by faulty third-party drivers mucking around in kernel mode. However implementing user mode drivers adds some new complexities if hardware manufacturers don't want to take a performance hit and from my experience not all hardware vendors are up to speed yet.

Driver Verification - this to me is the most troublesome and annoying issue right now with the 64-bit only version of W2K8. Only kernel mode software that has been certified in the MS lab is allowed to execute on a production boot of the OS. Period. Since I am writing this on the SoftLayer blog, I am assuming most of you are not selecting hardware and drivers to run on your boxes. We are handling that for you. But let me tell you it’s a pain in the butt to only run third party drivers that have been through the MS quality lab. Besides not being able to run drivers we have developed in house it is impossible for us to apply a patch from even the largest of hardware vendors without waiting on that patch to get submitted to MS and then cleared for the OS. A good example was a problem we ran into with an Intel Enet driver. Here at SoftLayer we found a bug in the driver and after a lot of back and forth with Intel's Engineers we had a fix in hand. But that fix could not be applied to the W2K8 64-bit boxes until weeks later when the fix finally made it from Intel to MS and back to Intel and us again. Very frustrating.

Okay, so now that you see some of the reasons NOT to use MS Windows Server 2008 what are some of the reasons it’s at least worth taking a look at? Well here are just a few that I know of from some of the work I have done keeping up to speed with the latest driver model.

Improved Memory Management – W2K8 issues fewer and larger disk I/O's than its 2003 predecessor. This applies to standard disk fetching, but also paging and even read-aheads. On Windows 2003 it is not uncommon for disk writes to happen in blocks

Improved Data Reliability - Everyone knows how painful disk corruption can be. And everyone knows taking a server offline on a regular basis to run chkdsk and repair disk corruption is slow. One of the ideal improvements in terms of administering a websever is that W2K8 employs a technology called NTFS self-healing. This new feature built into the file system detects disk corruption on the fly and quarantines that sector, allowing system worker-threads to execute chkdsk like repairs on the corrupted area without taking the rest of the volume offline.

Scalability - The W2K8 kernel introduces a number of streamlining factors that greatly enhance system wide performance. A minor but significant change to the operating system's low level timer code, combined with new I/O completion handling, and more efficient thread pool, offer marked improvement on load-heavy server applications. I have read documentation supporting claims that the minimization in CPU synchronization alone results directly in a 30% gain on the number of concurrent Windows 2008 users over 2003. That's not to say once you throw in all the added security and take the user mode driver hit you won't be looking at 2003 speeds. I'm just pointing out hard kernel-level improvements that can be directly quantified by multiplying your resources against the number of saved CPU cycles.

Alright, no need to beat a dead horse. My hope was if nothing else to muddy the waters a bit. The majority of posts I read on our internal forums seemed to recommend avoiding W2K8 like the plague. I'm only suggesting while it is certainly not perfect, there are some benefits to at least taking it for a test drive. Besides, with SoftLayer's handy dandy portal driven OS deployment, in the amount of time it took you to read all my rambling you might have already installed Windows Server 2008 and tried it out for yourself. Okay, maybe that's a bit of an exaggeration. But still...you get the idea!

Ok, the title of this blog may sound funny but you would be surprised how many phone calls I get about that very subject. Sure it’s not that specific case every time, sometimes it’s a software issue, other times hardware. But in the end not being able to access your server is the worst feeling in the world.

Enter KVM over IP. (Also known as Keyboard-Video-Mouse)

Yes boys and girls, this wonderful feature provided on all mid to high-performance multi-core servers can be your best friend in a time of need. While on a routine support call, a customer of mine stated the server was blocking not only himself but a lot of his customers. I kept a level head and told him it was no problem. He paused for a moment then let me know just how big a deal it was, while he was explaining I promptly used the KVM to login to his server and shutdown the firewall. All of a sudden he stopped talking and said “It’s working!”, “What did you do?” I explained to him how KVM works just as if you were hooking up a console to your server, and can be used even if your public Ethernet cable is unplugged. I went on to show him where it was in his home portal and how all of this was given to him for free. Also I explained the issue had been fixed from my desk without ever having access to either the public or private ports on his server. The customer had never heard of such a feature and was amazed at how easy it was to use.

The beauty of KVM over IP is it removes the one thing many server owners dread, not being able to be in the data center when issues arise with their standard connection methods (RDP, SSH). With KVM over IP we are giving the customer a solution to that problem. Via KVM you can login to the management interface card, which in most cases resides on an entirely different network, and within seconds you will have access to your terminal as if you were standing right there in the datacenter!!! Not only can you connect to your server, you can manually power it on/off and also reboot your server all within the same management screen. Beyond server access you can monitor temperature readings as well as fan speeds in the server. The KVM card is a HUGE tool in any Softlayer customers’ toolbox and one that we in the Operations Team use often.

Here at Softlayer we are always thinking about how to make your business easier to run, whether it be implementing global services such as CDN, or making sure our customers have basic access to their server in the event of a crisis. Since starting my career here at Softlayer and learning of the KVM feature I’ve made it a point to inform the customer of the KVM interface along with all features that are offered to them (and believe me they never stop coming!) so be sure and check our announcements page because you never know what we will come out with next!