Slashdot videos: Now with more Slashdot!

View

Discuss

Share

We've improved Slashdot's video section; now you can view our video interviews, product close-ups and site visits with all the usual Slashdot options to comment, share, etc. No more walled garden! It's a work in progress -- we hope you'll check it out (Learn more about the recent updates).

An anonymous reader writes "Python guru Jeff Knupp writes about his frustration with the so-called 'DevOps' movement, an effort to blend development jobs with operations positions. It's an artifact of startup culture, and while it might make sense when you only have a few employees and a focus on simply getting it running rather than getting it running right, Knupp feels it has no place in bigger, more established companies. He says, 'Somewhere along the way, however, we tricked ourselves into thinking that because, at any one time, a start-up developer had to take on different roles he or she should actually be all those things at once. If such people even existed, "full-stack" developers still wouldn't be used as they should. Rather than temporarily taking on a single role for a short period of time, then transitioning into the next role, they are meant to be performing all the roles, all the time. And here's what really sucks: most good developers can almost pull this off.' Knupp adds, 'The effect of all of this is to destroy the role of "developer" and replace it with a sort of "technology utility-player". Every developer I know got into programming because they actually enjoyed doing it (at one point). You do a disservice to everyone involved when you force your brightest people to take on additional roles.'"

100% inclined to agree. DevOps is not really about your best and brightest pure programmers, but taking all of your jack-of-all-trades guys who specialize in "making shit work" and allowing them to keep things working.

You forget about the ops part of devops. A lot of ops people need to be made to care about what is running on the boxes they are supporting. By knowing more about what is going on it can help then priortize the work which they are most experienced at, along with helping the "brightest pure programmers" understand why the cool solution they developed is a POS in production.

I have seen some companies have their developers given autonomy, with their own DevOps, mainly because it allows for what is needed to get granted. New subnet for lab testing? It is a lot easier to get a DevOp guy to configure the VLAN for it than to submit a ticket to a different organization that isn't connected at all, nor knows what needs done.

Of all the organizations in a company, dev needs the loosest reins (while still keeping separation so that the loosened policies don't allow for a security breach to compromise other departments.) The other department that needs autonomy is QA, because $DEITY knows what needs to be tested against.

So, having an autonomous DevOps means that the dedicated programmers have people that know what they want/need, and have the ability to get that.

In my experience, this does seem to work and work well in SMBs that are not just hiring H-1Bs or offshoring their entire dev department in toto. Larger companies, depending on corporate culture, not so much. Dev and QA should be autonomous. They have to be because that is where things get invented and bugs get squashed.

My experience on this is that you need 'DevOps' to run the development environments but you need production ops to run, eh, production.

DevOps people don't necessarily have good understanding of the underlying issues of the production environment. There are several issues revolving around this: Security, availability, scalability, etc. As an example is dev who can secure the application but not the platform (database, web servers and so on).

Good handovers between the two are the key for success. If 'DevOps' changes the development environment in a significant way this needs to be taken into account when moving the version to production. Oh, you upgraded the database in development environment? I guess that explains why RTP failed and we had to roll back.

Some changes may be very simple to do in development environment, while difficult in production environment. Need some extra disk and upgrade your database backend? Sure, just get it done in the development environment but there may some issues when doing that in production environment; you may need unacceptable amount of downtime, etc...

You're supposed to have both "Developers who do Ops" and "Ops guys who develop" in one team to do "DevOps".

If you're working in a place that's done "We'll just get the developers to do Operations" then they're doing it wrong.

I just started a new position where they had just figured this out and split the team into two. A couple of our developers trend toward the surly, and they would get a bit short with our customers (who are internal customers, but customers nonetheless) when interrupted. Which totally makes sense, by the way, because it kills productivity, but it was causing issues on all sides. Now we have this separation and process where our customers no longer contact us directly but fill out a quick report that autom

I think the idea is to *find* good people that already have interests and skills that encompass the union of the two, and supplement the "good developers doing development" and "good operation guys doing operation stuff".

To be honest, I think a developer that has no interest in infrastructure is a developer that can't design a scalable, supportable service (you need to know how the infrastructure works in order to effectively use it). An ops person that has no interest in programming is an ops person that

One of my old jobs was managing this transition, with a lot of performance analysis, reliability, scability and disaster failover / recovery work as part of it. At least in that environment, the majority of devlopers I worked with were highly unreliable wrt these issues. (And I successfully pushed for getting involved with the projects earlier - and to have access to the test resources associated with the development teams so that my folks weren't totally overwhelmed or bogged down with scut work - so that

That's one thing - any single item task is going to be broken down into sub-tasks and assigned to different people if not groups accordingly. And then those tasks may not be completed in sequential order. Then there's going to be one crunch-time for everyone when the last task is complete and things don't work. So it's a really nice thing if they can get one knows-everything-about-everything guru to complete that job.

helping the "brightest pure programmers" understand why the cool solution they developed is a POS in production

Absolutely. I was in a startup as a dev, and by necessity we were the operations guys early on. It was a financial service that needed to be running 24/7, and having to deal with "oh shit" moments while under the gun instilled the importance of writing software that was built for reliability, graceful failures, recovery, and error reporting.

None of this "brogrammer" shit or programmer "cowboys" or any of that other nonsense.

100% inclined to agree. DevOps is not really about your best and brightest pure programmers, but taking all of your jack-of-all-trades guys who specialize in "making shit work" and allowing them to keep things working.

This, right here. I inherited the DevOps job title, even though it is exactly what I've been doing for years now. I can go in, find a problem, test a simple fix, turn QA loose on that fix, and even with change management, I can have it implemented far faster than the devs, who might fit it into their next sprint if you're lucky. They naturally get informed and fit a more elegant solution in for the next release (and sometimes they leave my fix checked-in just as it is).

Meanwhile, while yeah in a start-up company the developer(s) had to play sysadmin too, all-too-often they don't really know much beyond the basics, and so you really don't want one, say, tweaking HugePages in sysctl.conf, or planning SAN or VM Farm expansion for the next web project, or lots of other things. Similarly, I refuse to dig any deeper in code beyond the simply Python tweak or the obvious fix/workaround, since I only know enough to be dangerous when it comes to all of the dependency chains, not to mention all of the subtle gotchas in all of the codebases I work with (why? Because while a given developer may only need to dork around with (or even just only a part of) one codebase, I have to wrangle multiple projects - time demands that I prioritize what I know about them all).

It bears a lot of responsibility - you have to know what the frig you're doing, because downtime==money, and stakeholders will have none of it. On the flip-side, you're given a lot of leeway when it comes to what you're allowed to do in order to keep the uptime flowing. For instance, I get priority, where I can call up a network admin, security admin, or whoever I need to put through a change as soon as safely possible. I can order-up (within reason) whatever CapEx I need to build up for the next release, project, or what-have-you. Of course, you have to justify what you do, and if you do something stupid it's your nuts on the chopping block, but overall it balances out.

IMHO (and little else), I've seen a lot of sysadmins able to step up to the DevOps plate, but very few developers that would be willing, let alone capable (most that I know prefer to write code, and not get their hands dirty with the business of playing server-monkey or wire-monkey.)

DevOps is all about creating dangerous conflicts of interrest.DevOps may be acceptable in a startup where there simply aren't enough people to separate the roles.As soon as there are enough people, the roles should be separated.

A devops guy is basically judge, jury and executioner of his own work; it's destined to fail.

And I'd even go so far to say that we need MORE conflicts of interests.

A software company is full of conflicts if interrests. You have the sharholders who want money, sales who care about release dates, customers who request a feature, devs who know that that feature will have unpleaseant side-effects that the same users would not accept and so on.

"Resolve" those "conflicts" by completly seperating them into different roles, and you have a company where departments will fight each other to the bone and management will be busy with conflict resolution instead of actual work.

You need to have people inbetween those branches who know how to make them work together.

Developer "D" is paid to create software; he must get code in production to get good reviews and keep his job.Operator "O" is paid to keep the production stable; he must keep unfit code out of production to get good reviews and keep his job.

These two roles work well with conflicts; D quickly creates code, O says "no go", D improves code, O accepts code. D gets a bad review but the company is okay.

When combined in a single person D+O quickly creates code, knows he should say "no go" but als knows he'll get b

100% inclined to agree. DevOps is not really about your best and brightest pure programmers, but taking all of your jack-of-all-trades guys who specialize in "making shit work" and allowing them to keep things working.

There is a conflict in the role of a DevOp though that you need to understand. In the Dev role you will be working on a project and to a deadline, and will just want to make "ad hoc" changes as needed for your application to work. As an Op role you have to say "hold on, lets make sure all those system tweeks have no other impact, apply them via reproducible deployment scripts, and maybe go back and take another look at your hooks into the alerting/monitoring system". As long as you and your boss understand

The problem is being a developer and running an infrastruture are both full time jobs. When you ask developers to run their own infrastructure things get neglected. When things get neglected things eventually break and they break bad. I've seen this happen when development should be focused on customer facing issues.

1. Multi-task. Never devote your entire attention to one task when you could be juggling 7.

2. Run Lean. Why hire a dedicated expert developer, a dedicated sysadmin, a dedicated DBA and a dedicated network engineer? Make one person do all those things (see #1, above).

3. Run 100% and demand 110. Who needs expansion room for when things inevitably go pear-shaped? Which Murphy guarantees even when you don't tempt him with #1 and #2.

4. Run cheap. Demand maximum expertise from the lowest bidder. There's always someone in a third-world country who'd be GLAD to do items #1, #2 and #3 for pennies a day!

5. Use easily measured things to determine employee effectiveness. Lines of Code, Time on Phone, stuff that's easily objectively measured, unlike less tangible things like customer satisfaction (what, you think we bother to ANSWER those silly surveys?), externals (like poisoning 6 downwind countries) or time to do the job right the first time. Use these as weapons to demand more of #1, #2, #3 and #4.

6. Subscribe to overpriced buzzword-laden management fads to assist in accomplishing all of the above.

Yeah, I remember the good old days, when there weren't many demands on developers, and release schedules were easy, and if you said to your boss, "can I have more time?" He said, "Sure, no problem my good friend, have a raise too!"

Yeah, I remember the good old days, when there weren't many demands on developers, and release schedules were easy, and if you said to your boss, "can I have more time?" He said, "Sure, no problem my good friend, have a raise too!"

This sysadmin/scripter/system architect/DBA role exists in virtually every company that has a core business other than IT or software development. Even in a very large multinational, there's more utility in having one "Mr Wizard" in each business unit than there is in having a room full of software developers somewhere far away from the rest of the business.

It really is a support role and it's more an outgrowth of system administration than it is saddling your brightest software guy with managing the mail server. Of course, it's possible to get stuck in that role because there's nowhere to go from there, but it's a niche that suits some people. If it doesn't suit you, then move.

It's also a distinct role from the "do everything guy" at a startup, because at a startup everyone is multitasking and as the startup expands, new people are hired to take on some of these roles. DevOps is a role in itself.

This "there is no role for devops in a mature company" attitude cries back to the age of isolated business units with isolated departmental goals, often where sales sells products that don't exist and engineering produces products nobody wants. The money to run the company will come from somewhere!

In short, developers don't want to dogfood, because that's hard. It's much easier to not challenge yourself with divergent ideas from what you and your brainfund coworkers cook up: after all, developers ma

I've worked at a Fortune 100 company, who was looking to add a DevOps team, because our development and our deployment teams weren't working together as smoothly as we'd have liked. The development teams didn't know anything about the hardware their (very hardware-specific) software ran on, and the hardware teams didn't know what parts of the software needed testing on new hardware.

Of course, it's ridiculous to ask the hardware guys to be present at all of the software meetings, and vice versa. De

And then they stop doing _any_ of the tasks well. They don't show up for planning. they don't document their code, because "it's self documenting" or "documentation is unrelable". They say "Just Google It" when most of what is on Google about the task is _wrong_ and written by people who aren't aware of the subtleties. They refuse to mentor, because it keeps them away from the meetings where they can soak up and interfere in _every single groups's projects_ by

What sibling said... I've seen companies that refuse to have a DevOps position, but yet hire "System Engineers" that basically do the same damned thing.

The biggest danger I've seen is in trying to silo such a position. I actually prefer having the freedom to whip up port channels on my own switchgear and having my own vlans/IPs to play with. I need free reign over an independent and complete environment for QA and dev use (so I don't have to put in a change request and then wait a week just to, say, add one IP addy to a NetApp SAN's NFS export ACL on some dark and early Sunday morning.)

I've done this job in a siloed company environment before, and quite frankly it sucks. You sometimes spend literal weeks waiting for one stupid VM that you could have cloned off yourself in less than 10 minutes, and configured in less than five more. I remember waiting almost a month while two different IT departments argued over how they would route a simple outbound rule over multiple firewalls whose path crossed the realms of their two departments... meanwhile having the devs wait alongside with me until the parties in question got done measuring their penises and found a solution (it took a pissed-off client, and a subsequently scared VP to threaten some jobs, which finally got them to STFU and work something out).

Devs and QA alike need someone who can quickly cut through the bullshit and red-tape, understand what it is they do (and more importantly, what they need and why they need it), and as someone aptly stated, "make shit work."

Seen this many many times before. Cheap companies that have lots of developers and are too cheap to hire experienced admins... or an IT shop that thinks they can have the IT guys program instead of hiring proper developers. "hey, you work with computers, you guys can all do the same stuff, right?" Wrong.

While I have known developers that can sysadmin, and admins that can program... they are the exception not the rule. Quality suffers when you force people into jobs they are not qualified for. Companies know this, and they simply don't care as long as the managers think they are saving money.

Seen this many many times before. Cheap companies that have lots of developers and are too cheap to hire experienced admins... or an IT shop that thinks they can have the IT guys program instead of hiring proper developers. "hey, you work with computers, you guys can all do the same stuff, right?" Wrong.

While I have known developers that can sysadmin, and admins that can program... they are the exception not the rule. Quality suffers when you force people into jobs they are not qualified for. Companies know this, and they simply don't care as long as the managers think they are saving money.

If you think the sysadmin who can program or the programmer who can admin a system is bad, you should have seen what happened when they gave Visual Basic to a subject matter expert (SME) and said, "You can program!" Agggghhhhhh!!!!!!!!

I have seen far worse. Full on applications written in Excel VBA by those SME's, MBA's and Finance (degrees? I am not sure). And those people that somehow thought that:A) they weren't shit andB) they were now programmers and the job was so easy, why do we pay you assholes...

I see plenty of this, but it rarely has me worried. You have to take these Excel / Access "applications" for what they are: they typically exist as job aids for single persons or small teams. Is it pretty or sustainable? No: if the author of the application leaves and something needs to be changed or fixed, no support org will touch it with a 10 foot pole. This is where the difference between risk avoidance and risk management comes in. Risk avoidance means shaking your head in horror, and removing Acc

Quality suffers when you force people into jobs they are not qualified for. Companies know this, and they simply don't care as long as the managers think they are saving money.

I think the problem is in mis-defining the role. A DevOps can write code at a junior level, but you don't want one doing anything more than simple modifications and/or workarounds. He can also do most of the infrastructure bits of IT at an experienced level, but you should always have him in close communication with the senior security admin, the senior network admin, the senior storage admin... this is so that he doesn't wind up going all cowboy on the infrastructure just to recover from an outage, or wors

Funny how my company is going in the opposite direction. They want to get developers out of the production environment completely, and have an ops team do all of the support and maintenance. I for one am glad to see this happening.

Full-stack developer generally refers to a developer who can code the full software stack - UI, middleware, and backend. Also lots of QA people can code - automated testing is mostly coding - and lots of developers can't test at all. Author needs a bit more real-world experience.

I think anyone who's developed and tuned systems at reasonable scale tends to have skills not just across the software stack, but across the infrastructure as well:

That means familiarity with the features and limitations of many different storage solutions (SAN, NAS, distributed fabrics of local disk nodes all with spinning and solid state, not to mention the Amazon flavors) , networks, bare-metal boxes, hypervisors, OS's, clustering solutions, a multitude of languages, databases (relational, document, NoSQ

Also lots of QA people can code - automated testing is mostly coding - and lots of developers can't test at all.

If my experience is anything to go by, then the QA people on average code about as well as the developers can test. Which is to say, everything from "decent," to "not at all," depending on how good the organisation was overall.

You don't force your brightest people to take on additional roles - that is the whole point of a devops team in the first place. Making developers argue about deployments and sending builds to QA and managing your GIT server and development and QA databases and managing your bug tracker is exactly what your developers should *not* be doing, especially if those scripts necessarily have to be in a different language than your application. Sure, your lead developers and architects work with the devops team to support them so they can in turn support you, but that's as far as the relationship goes.

The way we used to do it, where every senior architect is also responsible for all of those other functions (and has to take the time from his team members below him to help build all that out), is exactly how you stop architecting your software: your leads spend so much time trying to automate the drudgery they aren't improving the app.

They aren't improving the app because all of their brainpower is no longer focused on the *customers'* problems, but rather their own and their teams. That isn't a good use of their time. Hiring smart people who need to understand the application and its environment, but are good at scripting these other languages of automation, frees up your team leads to doing what they did before and do best: focus on the application and getting the team to produce the code that serves the customers' needs.

Because I'm busy solving the client's problems, and I have developers underneath me building databases, front end templates, integration platforms, CMS implementations and the works. I DON'T WANT them patching AWS cloud servers for Heartbleed because THEY SUCK AT IT. Likewise, I don't want an IT guy who just installs patches and walks away. My IT department acts like that, and because of them, I still don't have VMWare and test machines in the office eve

The problem that others are having with DevOps is that they seem to be defining it differently than you are. What you wrote makes sense but the scenarios people are complaining about don't sound at all like your definition.

The problem that others are having with DevOps is that they seem to be defining it differently than you are. What you wrote makes sense but the scenarios people are complaining about don't sound at all like your definition.

That's part of the problem, yes. DevOps started off with reasonably laudable goals: to promote a methodology whereby development teams and operational teams were tightly integrated in a way that made operational and deployment issues part of the development process: development would be driven by the need to deploy useful functionality, not just create it. That way, you didn't have a discontinuity where things were programmed, then someone would have to figure out how to actually deliver that bunch of code.

The problem which the author of the article references is that this often gets perverted from the original laudable idea of teams of developers and operations people working together, to requiring every single DevOps person being equally qualified to do everything, and then from there pushed even further to many companies creating DevOps positions where those DevOps people are literally doing everything, and not just knowledgeable at those things.

There is no question that a programmer that understands SQL or database architecture or storage systems or high performance networking or internetworking or virtual hypervisors is a more valuable programmer. They can use that knowledge to guide their development, write better code, and communicate better with the actual DBAs and network engineers and sysops and hypervisor admins. But when management types start to think that the best way to do things is to hire DevOps qualified people to just randomly do everything without any focus or specialization, that's when the myth of DevOps overtakes the reality of DevOps and begins to create real problems.

I don't honestly know to what degree that is pervasive in the industry: I haven't seen too many examples of it myself outside of certain high profile ones (the author mentions Facebook). If it is trending upward, I think its a bad trend. But to the extent that I see companies use DevOps correctly, as the glue-people to interconnect individual development, operational/deployment, and quality assurance teams, I think its a positive. But I agree with the article author that actually *replacing* developers, QA people, and operational people with DevOps people universally would be a Bad Thing. I just don't know if its actually really happening

The problem that others are having with DevOps is that they seem to be defining it differently than you are.

It's like agile... when it became hip to be agile, some people started reading thick books about it - at which point - the point is missed:)

DevOps is possible because you can rely on hosted services, like github, travis-ci, heroku (the whole push-to-deploy movement), EC2, AWS S3, Azure table Storage, and various other vendors like CloudAMQP, Heroku hosted postgres, etc...

All of these hosted services means that there is much less administration and maintenance, plus, these services can be provisioned w

Well I would counter that a DevOps team is needed even for customer-deployed or self-run cloud apps (where you're running the cloud in a data center instead of using services like NetApp or AWS). The jobs they do are useful there, in fact more so as staging DBs and environments for QA is harder when you're running all aspects of the cloud layer (though with customer-managed old-school desktop applications, QA is more responsible for their deployments, since testing installation instructions becomes part of

There's definitely truth to what he's saying but it cuts the other direction as well. Having your lead guru developer swapping disk drives on a machine isn't the best use of his time. However, I've also seen environments where the developers can't/won't/aren't allow to do the system admin tasks and wind up waiting around or being frustrated when their development systems have a problem. Likewise, with QA - I've seen developers that will just toss any old crap over the wall and expect QA to catch all of their bugs. And, developing tests is often software engineering, often complex software engineering that needs an experienced developer to establish at least the outline of how everything works.

Personally, I expect any developers I'm working with to have at least basic sys admin abilities and know how to setup/fix any other part of the stack they might touch. Those skills should be used when working with the dev systems and in establishing the base line for production. I would then expect that someone who is more specialized in those other roles to actually setup and run production and also be available when the developers get in over their heads on system admin, hardware troubleshooting, etc. In the same way I would expect a systems admin to at least be able to write a script to automate something and not go running to the developers for everything.

For test development, I always like to set groups against each other and develop the test suite for each other's code. Most people are a lot more comfortable and eager to break someone else's code than they are their own.

Most people are a lot more comfortable and eager to break someone else's code than they are their own.

Not me. I'm just as merciless with my own code as I am with others' when writing tests. Proper testing involves taking on the role of the malicious agent who is actively trying to break the code, feed it bad inputs and generally muck up the works. If the code passes those tests then it stands a much better chance of rolling with the punches in the real world of production.

Deveops types aren't the kind of people to be crawling around under desks or helping directly to push for a release milestone.

They're the go-betweeners, sort of a cross between senior sysadmin and development project management assistant. They are the internal toolsmiths, depending on the blacksmiths to produce effective metal so they can hone the tools for the carpenters' needs. They are broadly skilled and know how to at least muddle along in both a developer and a sysadmin job, but prefer the big picture

A colleague said to me once, "You're the only VP I've ever known who installs equipment." I'm always amazed at how out of touch "senior" people can get. And besides, when we get new toys I want to see them working NOW.

I had a job where I did everything once, wrote a full-blown ERP system for hundreds of users, all by myself. Everything. Though I was salaried, sometimes I worked whole weekends, or to 2 in the morning - not because I had to, but because I wanted to. No politics, no being just a cog in a machine, no project management, no BS. Just me and code, giving people what they needed and making their jobs easier. It was my dream job, my first and best job, and I've never had anything like it since.

This DevOps movement the author speaks of... I've never seen it, not in all the years I've looked to find it again. He may complain that it's bad, bad for the industry, but I would take it in a heartbeat.

I was going to say something similar. There are a lot of people who don't want to be kept in a box, doing only one thing all day. I used to hire those people specifically - I found it worked well for our culture. Plus, I personally get bored with too much of the same thing. I love coming up with a cool SQL trick that cuts the cost of a query by 99%. It is fun to watch something complete in 2 seconds that was taking 15 minutes before. But I wouldn't want to do that all day, every day year after year...

another thing where your mailserver is falling over and a major customer is screaming about how they were promised a major feature today and they're going to walk if they don't get it... and your jack-of-all-trades is demanding that you put your priorities in writing to cover his ass when one of the two doesn't get taken care of.

I am primarily a developer but I also like to understand the big picture, including software design and UX but also system administration, infrastructure, hardware architectures, and everything else that *directly affects the software I develop*.

Deep understanding of the big picture is key to developing quality software, IMO. You need to understand what comes ahead of you (requirements, business needs, etc) and where your work is headed afterwards... The best way to understand it is to wear these hats from time to time or have previous work experience in those fields. When recommending candidate for developer positions, someone who has system administration experience is a bonus.

Yes, many days I need to take on multiple hats and switch gears as shit comes up in prod and I need to fix a config on production servers or assist whoever has the hands but lack the knowledge. That's the start-up culture I guess, even though I work for an established 100+ year old company.

And regarding the "Totem Pole" argument in TFA: That is utter BS. I've seen many so-called "developers" unable to perform basic DBA or QA jobs. That is why most large company employ specialist in these positions, to cover the fuck-ups of the developers.

I've seen developers who's first reaction when a complex SQL query is taking ages to run is "I will call my old pal in XYZ dept who is a DBA" before even running an "explain" query to debug their index-less octo-way join with 12 sub-queries.

I worked for over a decade in embedded stuff, mostly Linux kernel development. I didn't do a smidge of database coding the whole time. It just wasn't a factor. On the other hand, I did learn assembly language for five different architectures, found locking bugs in glibc, binary-patched the running kernel for a production fix because the client didn't want to take a new release, and a bunch of other interesting stuff.

And now I'm working on a different project that uses databases heavily, so I'm picking it

The point of devops is not to take jobs away from developers. The point of devops is to provide an interface between system administration and development. Development and system administration have always been at odds with each other - system administrators not really understanding or caring how the application works, and developers treating the systems as an infinite resource pool with no real rules or resources past "does my code run?"

The sole purpose of devops is to ensure efficient operation of the infrastructure in a way that allows for repeatable deployments and controlled versioning, and that also includes system software such as operating systems (sysadmins benefit too because they no longer have to do one off deployments of OSes).

This criticism strikes me as woefully underinformed as to what devops actually does, and I'm wondering if the author of this is a developer who is upset because devops is forcing them to actually use the software lifecycle properly rather than just doing cowboy deployments and hoping they work.

Devops is even more than this, it also means applying common change management that is common in source code development to operational configuration, using tools like chef or puppet. It's scary how much "cowboy configuration" there is out there, and yet in the programming world, "cowboy coding" is frowned upon.

The configuration "Developers - DevOps - Operations" makes sense. Unfortunately, a lot of companies just remove everything but DevOps from the picture because having dedicated developers and admins around would just be redundant.

I'm in such a company and its a hellhole - you can get emergency calls at any time of day because you're responsible for the infrastructure but that's not reflected in your pay because hey, you're just a developer. Also, keeping the infrastructure running flawlessly is not suppose

I've been a developer, a sysadmin, and a devops guy. And the number of developers I've had to gently (and sometimes not so gently) say "stop doing that, you're crashing the server!" is way more than I am comfortable with...

The best developer is one who puts stuff together that 'ops' people (users, admins, etc.) can work with. And the best way to get such developers trained is to give them some experience on the other side of the fence. Yes, in a large organization there is going to be less crossover. But its still a good idea. Some people won't like being admins. Some will really take to it. Its up to management to properly allocate resources and keep their people trained and familiar with adjoining organizations needs.

If you absolutely don't want to do any administration tasks, that's fine. But its a rare developer who doesn't throw a fit when management takes their admin/root privileges away on their own workstation.

It's legitimate for a developer to throw a fit. They should have admin access on their local box. There's an insane number of plugins, libraries, and frameworks they may need to install or test to determine if it's the right fit for their deliverable.

A problem I have run consistently around is that developers rarely understand system administration, and operations in general. It makes their software suck a lot more. This is even more true with the Java-crowd, many of which cannot even use a commandline. The more this gets fragmented, the more people get specialized, the more problems arise in architecture, design and implementation, up to and including software that cannot even be deployed because of misconceptions on the developer's side.

I essentially have this kind of role within my organization. I design, develop, deploy, and support small to mid-tier systems (e.g., the planning system for a $XXXmio/yr global department, with 300+ direct users) while being one of my own customers, as I am actually a business planner (by role) as opposed to developer. I develop systems as a way to do my "day job" much more effectively. Typical tech stack would be Excel UIs, PostgreSQL data store, and whatever else I need in the middle (e.g., nodejs, tomcat, redis, whatever).

What I've found is that, in general, doing the right thing the "right way" is not worth the cost compared to doing the right thing the "wrong way". By definition, in either scenario, the right things is getting done. What most pure developers utterly fail to understand is that in trying to do the former, there is an overwhelming tendency to do the wrong thing the right way instead.

This is because, as Fred Brooks pointed out long ago--and as the "lean startup" movement is re-discovering today--for any non-trivial novel problem you cannot know in advance what the "right thing" is until you've actually tried to implement a solution. Brooks stated this understanding as the need to throw away the first try, and the lean startup movement is essentially defined by a corollary--you have to figure out how to try cheaply enough that you can afford to throw it away and try again (and again, and again if necessary), while progressively elaborating a robust definition of what the "right thing" looks like by using those iterations as experiments to test hypotheses about what the "right thing" is. Doing things the "right way" usually costs so much in time if not capital that you simply can't afford to throw away the first try and start over, or you cannot complete enough iterations to learn enough about the problem.

Now, I'm not saying that you should be totally ignorant of software engineering best practices, design patterns, etc. What I am saying is that there is a limit to how effective you can be in reality if you live purely within the development silo. Having a "DevOps" role (granted, self-imposed in my case) has been one of the best things that's ever happened to me as far as making me a better developer, right up there with the standard oldies like writing your own recursive descent parser and compiler.

In short, it is commonly-accepted wisdom among programmers (for good reason!) that you are more effective if you actually understand the technology stack down to the bare metal or as close to it as you can manage (even if only in abstract-but-helpfully-illustrative examples like Knuth's MMIX VM), and that this understanding can only be gained via practice. It should be obvious that the same is true in the other conceptual direction through deployment and end use.

The difference is between developers knowing the operations side and being the operations side. Developers need to know the operations side to know how to write software that Ops can install and manage. And they should be involved in the development environment and installation in the testing environment so any gotchas can be addressed quickly and the developers know exactly what happened and can go back and make sure it doesn't happen again (especially in production). And of course when things really go pe

I suspect there's a bit of a definition issue at play here (with all fault apparently being on my end, given some of the other comments in this discussion). In my mind, DevOps roles are such only if the "Dev" and "Ops" parts are connected--I.e., you manage operations for the software that you've developed. I agree that there are rapidly diminishing or negative returns otherwise. E.g., if you write some Nodejs web services on Monday and troubleshoot MS Exchange/ActiveDirectory integration issues on Tuesda

If devs have to LIVE with the consequences of their bugs or race conditions or misfeatures, by having to manually fix up customer accounts, or reboot the server, or just click through too many screens, then it really focuses their minds on what are the important bugs to fix, helps them triage more effectively, helps direct their coding energy in the right place.

PS. This is only for devops work on the systems running YOUR software. If course it doesn't apply to what most people

> Have you ever noticed that companies locate their research divisions away from the day-to-day operations divisions? It is to keep the timeframes separate.

No, it's turf building and budget protection. By segregating the developers from devops, devops can _hide_ their resources and keep them sequestered from developer requests. And putting the systems into a "requests go to managers, and only then to devops" makes the managers vital to allocating resources. It can protect their team from excess pecuniary

Anyone else remember a day when this function was split? Back in the early days, we had Systems Analysts, kept in a separate room to the programmers.

DevOps,,. we've seen this sort of thing before. End result is that we get less specialists, and things become harder to fix when they break. But, multi-skilled people are the way things are going. And have been since... well, Michaelangelo was a sculptor who was asked to paint. And I have some of his sonnets, somewhere.

you used to carry your stack of punch cards to a Demi-priest, who passed them to the Priest who was the only one that was allowed to touch the computer.
Funny, how even today that role still exists.. it's called a DBA.. these jerks won't let mortal developers near a database, even though the dev writes the script to be run against production, and DBA monkey just hits Run for you.

Devops is not killing off developers. What it is doing is combining the jobs of a traditional Release Engineer and a traditional Linux System Administrator. It's right up my alley, actually, so I am looking to move on for exactly such a position.

"Somewhere along the way, however, we tricked ourselves into thinking..."

So: when a good idea is implemented poorly, then bad things happen. Why is this news?

'DevOps' isn't killing the developer; people who are abusing developers are killing developers and using [place idea here] as an excuse. If you focus on 'DevOps', then you're going to throw out an idea and do nothing to prevent people abusing developers and using [idea n+1] as an excuse.

New code is not perfect on creation and developers need to be prepared to swing the axe, scrap stuff that doesn't work and start off different approaches.However in production it's better to make very gentle changes to a unstable house of cards even if it is utter shit held together by chewing gum and string.To work in both you need to be able to switch between mindsets because they are really very different jobs. Someone working in both is not ideal. Even if you are very good at both there is a strong te

You do a disservice to everyone involved when you force your brightest people to take on additional roles.

This sounds as if the author is saying developers are brighter than other people. Well a few may be, but when you look at most of the dumbass bugs that appear (not to mention the spelling mistakes, tortured logic, crappy coding styles, and mistaken ideas of what constitutes "good") I really can't see that being the general case.

As it is, I feel that it does developers GOOD to get them into a position where they see apps and O/S's from the other side. After all, these were developed, too. So all you're aski

In essence DevOps is an approach to not only allow continuous integration, but go further and allow "continuous deployment". This is necessary to allow software to follow user needs more quickly. Nowadays in many large companies there is some software governed by the IT department and many small tools are ungoverned derivatives of other tools implemented by some management person in Excel. A lot of company knowledge resides in these software pieces. And companies rely on such tools. However, those tools are

Would agree but I don't. Primarily, to design en develop systems well you should have a full view of all its influences across your organisation. A well designed system should intentionally consider all aspects well, against the business strategy.

Having worked for huge organisations, I must say systems are rarely designed well. Their influences across all disciplines are hardly ever fully considered. From one side I agree that one should avoid paralysis through analysis and that eventually some programmi

There are extraordinairly few developers that have a decent understanding of security principles, networking, or any one of a thousand things that the sysadmin does every day. The mistakes I've seen well educated developers make.

"You do a disservice to everyone involved when you force your brightest people to take on additional roles."

I'm not sure that's accurate. Programming skills are pretty important today and most people, including myself, learn to code so that their current skill set (and market value) can be enhanced. It used to be that if you need an application, you'd have to buy one or hire a contractor to build one. So, this route has some problems associated with it besides cost. For example, if what you buy breaks, it's harder to fix without calling the company that made it. If you work in telecom and need a call center wallboard, you can either buy one or make one. I chose to write my own with Debian, PHP, and Apache. And it works just the way we like it, we have full documentation on how to rebuild it if needed, it's an inhouse solution that was built by "DevOps". There was a steep learning curve to develop the skill set to do this - it took a few months of hard work and learning. But it was totally worth it.

If anything, the existance of DevOps is totally natural and was born from necessity. If I were a full time programmer, would I be bothered by this trend? Maybe.

Nice when you get the time to develop every system from scratch, and don't have multiple departments saying "We need this now". And actually meaning it.Then having to ensure that what you've put in place stays up with a 24x7x365 uptime requirement, and recoverability to the last transaction.Oh, and hey, this thing that some department has purchased because they couldn't wait for the system to created, it only runs on another version of Linux (or on Windows with SQL Server!).. They need that put in there t

Several problems with the original article. Jeff makes the assumption that Developers are at the top of a totem pole and all others are below this developer hierarchy.
This is the problem with a lot of younger developers, most who are jumping into development with no structural background in the operating system or real production operations. This premise leads developers to think they are superior to everyone, rather than focusing on building applications that must run in an operational context of being well-behaved in the environment they run, and allow for maintenance work of the operations environment. This is why you see anecdotal articles coming out that talk about the frustration and friction between developers and operations people.
The truth is, most developers will be better developers if they learn a little more about how their app will work in a production environment. They will build better applications because they understand the environment better. Your architecture should be loosely coupled, that is, it should not stack up on a single server but rather be broken up into smaller components that can be built for scale across multiple machines and be able to fail over to redundant systems. Ops people will be more than happy to help get you there, but you need to think about how your app will fail as much as how it will work properly.
The developer also needs to stay away from dictating specific languages -must- be used to run operations. Rather, they need to understand that operations is best suited to a polyglot of different kinds of tools and languages, using the right tools and wares for the job, not being dictated in some kind of nanny state that says you must use this language or that for operations and all others are to be thrown out. All too frequently startups get hung up on dictating operations people must use Python or Ruby or the ops dude is a total loser. This is the kind of arrogance that shows how little most developers really know about operations. Any of the more popular tools should be available to the DevOps person so they can get things done, and developers should stay away from being nannies.
In classic UNIX operating environments the system administrator is/was god, followed by an allowance to use the system granted to developers, database admins, etc. Applications have always been guests inside most Linux/UNUX-based systems, even Windows treats applications for what they are, programs that run inside the operating system. This has always been the premise of modern UNIX-based and even Windows-based environments. Ops people follow this naturally, but typically run into developers that don't want to follow the rules.
Ops people expect you to write a well-behaved application. Developers expect an operations environment that runs their application without getting in the way. These two goals can be easily met if both sides work together and allow each side to exploit the system, but everybody has to allow the freedom to make things work best for the goal of building better systems.

Horrible assumption that you think developers are top-dogs, Jeff. I've seen many cases were DevOps role FAILED for competent developers because, in the cases I've seen, this is true:

* Your smartest developer may be just that when it comes to language, software architecture and platform development, but their operating system, networking, hardware infrastructure knowledge + background is not even hobby-shop at best.

* They've always had an ops or engineering crew to throw their code at, figure out how to integrate it, and NEVER had to support it.

* Ego problems thinking they are 'above' remedial automation --- which most of the time doesn't involve a real development language, just scripting.

Out of those two things alone, I've always heard the: "Well we need a sys-admin/engineer now because we are spending more time trying to manage systems, not really sure how or what to automate, and it's really taking time away from me getting back to the kind of development I, as the developer want to do." Which is the polar opposite of the two points I mentioned above.

A working DevOps group should be an amalgamation of the Dev team and Ops teams.Trying to forge one team where everyone knows everything simply sets a level of mediocrity; you can research so far down an avenue in a given time, and get only so good at it.I've learned from the ground up (i.e. electronics, basics of VLSI, board design, basic OS design, all layers of the stack programming), and went from there on to system admin. Then did a stint as a developer using the knowledge that I had from my earlier history, and found that the sysop area of my knowledge atrophied in the detail (and the devil is _always_ in the detail) the more I concentrated on being a better dev.Went back to more of the operator/business side of things, and lo and behold, the more I go into systems and how to put together a proper reliable, recoverable infrastructure, the more my dev is atrophying. I'm half management these days, which means the ops side _and_ the dev side are both atrophying. The guys that do it in a dedicated fashion are more familiar with the latest tech than I am..

You can be a jack of all trades.. But I seriously hope a company doesn't rely on you to get them out of trouble when the fecal matter hits the fan.. If you've been spending most of your time developing, with the nod to tuning the servers so your app runs better, you're not likely to have been able to put the time in to develop the wider infrastructure to support things going fubar, or had the time and concentration to really work out what is likely to get you.

Having a few people marked as DevOps would be useful when you need to populate a middle ground.. They can work with both dedicated ops, and dedicated dev to ensure that scalability is baked in, and resilience is baked in to the apps. When it comes to ensuring the boxes are kept in tidy order for everyone, and get to be able to recover from the smoking ruins.. That's where the dedicated ops shine. When you really want that app to do something really slick, that's when a dedicated dev shines.

Small scale, a DevOps person would work. The larger you scale, the less appropriate it becomes (as the only solution; a big company with the techs being solely a DevOps team would scare me).

Exactly. Most developers are incapable of producing code that anyone would ever want to use.

In multiple companies for which I have worked, it was known who was capable of producing code that could actually be shipped and who could not. Management highly valued those who were capable of producing shippable problem free code.

That, though, is talking about the development environment itself. Yes, there the developers should be in control of the machines. DevOps, though, is about having developers actually doing operations tasks in the production environment. That's a bad thing, because what developers are good at is very very bad in production. You don't want developers alpha-testing code and fixes where a failed test brings all of your customers to a screeching halt while the developer does a few more iterations to get his fix