Random thoughts and technical bits

Death of the sysadmin and birth of….

I started my career as a sysadmin.. I didn’t want to spend all day sitting in a chair writing applications. I wanted to touch the hardware. I am a firm believer that every sysadmin is a control freak to some respect. They love how the machine obeys them. They enjoy telling users no you cannot and figuring out ways limit access. The essence of every good sysadmin is the innate need for improvement. In the early days of my career I was exposed to systems administrators who had hundreds of shell scripts they had everything automated. As the years past these older sysadmins seem to be replaced with younger admins who had been raised in an easy world. They were used to clicking next to install applications and things that just work… (hence the appeal of the iPhone). I am all for simple and easy gadgets and bringing computers to every old persons life. The relative ease of the solutions have made life a little too easy for us.

Cloud

Then this crazy thing happened…. the cloud. Amazon brought the easy button to server deployments. Some embraced the ease of the solution, others liked the agility. What ever your motivation for using AWS they have changed IT again. Everywhere I go business units want to know why it takes so long to deploy a server. They want to know how to create their own AWS cloud. People every where have been deploying operating systems and getting IT done without systems admins..

The Auto Industry

When the auto industry first started Henry Ford and his engineers would assemble a car from scratch. Everyone working on the car understand each component and how it worked. They understood the flow of assembly to make the car. Each of the assembly guys could build a car from scratch or design a car. As time past demand increased for the product and Ford had to increase his agility to create cars. He hired workers to build cars, assigned them specific roles with rote tasks. These workers would do the same task over and over again. This provided a few advantages: first they got good at the task and they did not need to know how to build a whole car. It also introduced some challenges: if they missed their task due to human error failures were introduced. Eventually humans workers were replaced with robotics. This reduced the errors and increased the cost. It also allowed Ford to build a lot more cars. Not all the jobs went away they just changed. Workers were replaced with robotics and automation engineers. The people working on the cars had no idea how to build cars they just kept the robotics working. Every other car manufacturer followed suite to compete. Auto manufacturing plants became huge, downtime cost millions of dollars. Massive amounts of money are spent to ensure the plants keep running.

What lessons can we learn from Auto Industry

Having highly skilled humans build the cars worked great

Having architects design cars then hand off work instructions to workers introduced a lot of errors

Having automation reduces errors and requires workers with a new skill set

It is not required that the people keeping the automation running have an understanding of the product, they just need to understand the automation

The cost of automation will force a centralization of building cars

As manufacturing became centralized downtime became a critical issue

What does this have to do with sysadmins?

Thanks for bearing with me this far. If your still reading and wondering why I wrote this article let me explain. I want to suggest that the world is changing for systems administrators and as control freaks they don’t really like it. AWS has created a golden standard we can deploy the system in minutes why can’t you? If you have not faced this question you will soon. Every shop wants to have AWS.

Every shop wants to have AWS but do they need it?

AWS has a very specific business model. Deploy base templates for customers then step away and collect cash. It’s a great model. Functionality of the virtual product beyond being powered on is 100% your problem. AWS ensures uptime of power and networking. You still have to do a lot of work to make that server deployed in minutes usable. AWS has saved you the time of procurement of hardware and working with silo’ed team to get a server in place, but you don’t have a money-making machine until you install your product. What is it about AWS that you really need? I suggest it is not agility instead it’s less hassle. AWS provides you freedom from people who seem to create never-ending road blocks while doing their job. Yes, I am looking at you security team. Yes, I am looking at you server deployment team. Yes, I am looking at you…

Why is IT so hard

IT is hard because it’s never the same. In my career I have rarely seen the same request twice. If your business is netflix and you have three types of servers then automation makes sense. Most IT shops are not netflix’s every single business unit wants to drive IT choices and so we get a spaghetti mess of IT. IT is hard because the business unit wants to drive technical choices instead of business requirements. Many years ago we had a business unit demand that their new workflow be built in Sharepoint, forget the fact that we were a linux shop with no sharepoint. So the lesson is:

Business unit’s stop messing with IT. Bring your needs well-defined to IT and let us implement it. Trust us to do our job it’s why we cost so much.

Why do Menu’s exist

Restaurants have menu’s for the following reasons:

To limit customers options – they could not possibility have all ingredients

To help customers make choices – if left to their own customers would become confused by the options and leave

Create standard workflows and realize cost savings

Give the customers illusion of choice

Why doesn’t IT have a Menu…. here it comes the ITIL service catalog. Most service catalogs are too technical and don’t represent what the customer really needs. What the customer needs is a service which is normally a lot more complex than a single server. They have a project.

Project

Yep that word again project… it’s so important we have a certification and role who manages it. Business unit’s rarely want one more netflix streaming server.. they expect IT to handle that if needed. They want to create a whole new business and that requires a project. Our menu really needs to be a project menu not a server menu. We need to stop offering the business unit separate components of our offering or they will keep getting into our business. We need to provide the business unit the correct choices that keep them away from dictating technology.

Death of a sysadmin … birth of a ..process engineer

So now that I have ranted for too long what is the future of systems administration. I think we need to become process engineers. Very few people are going to understand the whole product. More will administrate from a automation console rather than logging into a server. How do we re-tool for this change? I have a few suggestions:

Learn to examine process. Do something manually first. Document the process in extreme detail, use a process diagram. Critically look at your process diagram. Do you see how many manual processes you have? How can you automate them.

Help customers standardize, learn the language stop jumping to technical solutions with your customers. Focus on their needs and requirements allow the technology to be a black box.

Develop standard methods for documenting and ingesting new projects… create a documented process and follow it.

Automate everything you can, develop solution with the automation mind set. How would I do this if I had to deploy 100 servers instead of two.

Ask your self does this process, technology or choice scale up? If I had to increase the amount of these by 1,000 would this process work.

About Author

Joseph Griffiths is a virtualization focused solutions architect who works with complex cloud based solutions. He currently holds many IT certifications including VMware VCDX-DCV and VCDX-CMA #143. This blog represents his random technical notes and thoughts. The thoughts expressed here do not reflect Joseph’s current employer in anyway. You can follow Joseph on Twitter @Gortees