Process kills developer passion

Best practices sound good in isolation, but they can suck the life out of developers.

The other day, at lunch, I had a bit of an epiphany. I also had the pulled pork, but that’s another story. In any event, something came into clarity that had been bothering me below the surface for a long time.

Over the past few years, the software industry has become increasingly focused on process and metrics as a way to ensure “quality” code. If you were to follow all the best practices now, you would be:

Doing full TDD, writing your tests before you wrote any implementing code.

Requiring some arbitrary percentage of code coverage before check-in.

Having full code reviews on all check-ins.

Using tools like Coverity to generate code complexity numbers and requiring developers to refactor code that has too high a complexity rating.

In addition, if your company has drunk the Scrum Kool-Aid, you would also be spending your days:

Generating headlines, stories and tasks.

Grooming stories before each sprint

Sitting through planning sessions.

Tracking your time to generate burn-down charts for management.

In short, you’re spending a lot of your time on process, and less and less actually coding the applications. I’ve worked on some projects where the test cases took two- or three-times as much time to code as the actual code, or where having to shoehorn in shims to make unit tests work has reduced the readability of the code. I’ve also seen examples of developers having to game the tools to get their line coverage or code complexity numbers to meet targets.

The underlying feedback loop making this progressively worse is that passionate programmers write great code, but process kills passion. Disaffected programmers write poor code, and poor code makes management add more process in an attempt to “make” their programmers write good code. That just makes morale worse, and so on.

OSCON 2011 — Join today’s open source innovators, builders, and pioneers July 25-29 as they gather at the Oregon Convention Center in Portland, Ore.

Now, I’m certainly not advocating some kind of Wild-West approach where nothing is tested, developers code what they want regardless of schedule, etc. But the blind application of process best practices across all development is turning what should be a creative process into chartered accountancy with a side of prison. While every one of these hoops looks good in isolation (except perhaps Scrum …), making developers jump through all of them will demoralize even the most passionate geek.

I don’t have a magic bullet here, but companies need to start acknowledging that there is a qualitative difference between developers. Making all of them wear the same weighted yokes to ensure the least among them doesn’t screw up is detrimental to overall morale and efficiency of the whole.

Now, this may sound a little arrogant: “I’m an experienced developer, I don’t need any of these new-fangled practices to make my code good.” But, for example, maybe junior (or specialized) developers should be writing the unit tests, leaving the more seasoned developers free to concentrate on the actual implementation of the application. Maybe you don’t need to micro-manage them with daily updates to VersionOne to make sure they’re going to make their sprint commitments. Perhaps an over-the-shoulder code review would be preferable to a formal code review process.

And as an aside, if you’re going to say you’re practicing agile development, then practice agile development! A project where you decide before you start a product cycle the features that must be in the product, the ship date, and the assigned resources is a waterfall project. Using terms like “stories” and “sprints” just adds a crunchy agile shell, and it’s madness to think anything else. And frankly, this is what has led to the entire Scrum/burndown chart mentality, because development teams aren’t given the flexibility to “ship what’s ready, when it’s ready.”

Unless the problems I’m talking about are addressed, I fear that the process/passion negative feedback loop is going to continue to drag otherwise engaged developers down into a morass of meetings and metrics-gaming.

Oh, dear! Sounds like you’ve been through some management-mandated Scrum transition–the kind that added practices but missed the point of Agile.

http://www.xxcoder.net Stray

“But, for example, maybe junior (or specialized) developers should be writing the unit tests, leaving the more seasoned developers free to concentrate on the actual implementation of the application.”

Really?

Personally I love TDD because it kicks my brain into figuring out the corners and the complexity when I write the test, and then I get the benefit of all that juicy brain activity when I’m writing the code.

Obviously I’m talking about a one-or-two-failing-tests at a time cycle. So this process has a really short reward-for-effort timeframe.

That’s what the ‘drive’ bit in TDD means, surely? That the tests are the impetus for development, not just some PIA bit of housekeeping that you can palm off on the junior devs. If anything in my team it’s the other way around – I might write some tests, talk with the junior dev about what I learned in doing that and then get them to implement the passing code.

Yes, tests have a bunch of benefits – coverage lets you refactor without messing everything up, good tests provide awesome documentation for the next dev etc – but the primary benefit, in my workflow, is the way tests make me focus on the requirements, and really flag up weak points in my intended architecture before I’ve implemented them in the code.

The other thing is that the tests might have taken 2 or 3 times longer than the (working, test covered) code… but really we should be comparing test-effort with the bugs that didn’t get written and missed and then hunted down until 4am…

But yeah – I agree that development-by-numbers is a sucky process! The rest of my process is, joyfully, based on ship-what-works-when-it-works. A new feature that can be added in an hour makes it out the door that day. And because we do TDD we very rarely ship bugs, and when we do we fix them quickly, so the client is happy to have as many updates as we can ship.

The basis for that is of course trust between the client and the developers – they know we love our project as much as they do.

Meetings can DIAF. 3 or 4 a year is our limit.

George

“I’ve worked on some projects where the test cases took two- or three-times as much time to code as the actual code, or where having to shoehorn in shims to make unit tests work has reduced the readability of the code.”

I was with you up to this point. Writing good quality unit tests is a great investment that will pay dividends over the lifetime of the product. Sure it might take a bit of time that you would rather spend hacking things around a bit more but, in general, you will be glad they are there.

Oh, and in my humble experience, part of the value of unit tests is that they make the developer think about issues like extensibility, coupling, cohesion etc – more stuff that developers will be glad of over the lifetime of the product as and when changes need to be made. If those are the kinds of things that you need “shims” for then maybe that should tell you something about the code in question…

Canuck

Yeah, there’s nothing like management enthusiasm to kill a great bottom-up idea like TDD. The first rule with any best practice is stay well clear of the 80/20 line.

I have found TDD incredibly effective for developing everything from finance web apps to XML aid-data schemas. The greatest benefits don’t show up until you’ve been through a few releases, and you can still make substantial code changes as fast as you did during the first month of development.

OTOH, it would be brain-dead stupid to go past the line where writing tests is easy. If the last 20% of the tests take 80% of the effort (shims, databases, etc.), then just skip ’em.

I sometimes start with just a few tests, to allow rapid prototyping. If you focus just on writing tests for bug fixes and new features after that, you get almost all the benefit of TDD with only a fraction of the effort.

http://www.dalmenyclose.com Ross Hall

“companies need to start acknowledging that there is a qualitative difference between developers.”

Agreed.

However …

Enforcing standard processes gives inexperienced developers a framework in which to grow, reins in the developers who are a little “loose” with discipline and ensures any problems that arise down the line can be found, assessed and corrected later.

So yes, overly burdensome processes and the idiots who invented them should be asked to leave the room.

Those who decline to participate should join them.

Nathan D. Ryan

While I sympathize with your sentiments as a whole, there’s nothing wrong with a two-week burndown chart, which is supposed to be about work remaining, certainly not time tracking (which I consider antithetical to agile practice).

zebr

I agree completely with the general premise — software engineering all too often focuses on how to marshal programmers’ productivity via buzzwords…

“But, for example, maybe junior (or specialized) developers should be writing the unit tests, leaving the more seasoned developers free to concentrate on the actual implementation of the application.”

But this is a surefire way to kill junior developers’ passion.

The way to solve this problem is *not* to give junior developers the dull, tiring tasks. How do you become a seasoned developer? Not by writing endless unit tests. You would lose any fresh talent that instant.

Jerry

I hear what your saying, and it’s good to hear this coming out of the development field. I’m a support technician, and I’ve had marginal experience collaborating with developers. Unfortunately, those I have worked with seemed nice enough at first, trusting me to carry-out tasks wearing big-boy jeans, but then later treating me like I should be wearing diapers. What I like hearing is that word “passion” because I use it sparingly; only for things I truly love. Your right, there’s a problem. It’s about perspective; skewed perspectives. Technicians who look up to developers are a rare breed as I see it. It only gets worse if developers are demoralized by micromanagement issues. Something tells me this is a growing trend, and even worse, I’ve seen it coming from management who claim they don’t have time to micromanage. It seems there’s a contradiction in terms here. Across the board there are failings to recognize a key factor: Ultimately the clients are the one’s who suffer most, when all of our groups have trust issues. That’s what it’s about. I have a friend, mentor, programmer-developer…Co-Founder of a successful software company who once told me: “Every issue is a management issue.” I argued with him on this, because I have so much passion for clients and went out of my way to address every party’s concerns and issues that I could; finding much turmoil and aggression throughout our groups. Ultimately, he was right…management controls morale, and they do so through trust. When they stop finding time to address issues personally, choosing to implement procedures and policies to reign in poor quality, they’ve handed their job over to….uh…”A Nobody?” This may sound skewed to many folks, but I’ve had epiphanies too, and watched them get mangled via pride issues up the chain of command. What seems logical has no coherence these days, and it’s because policies and procedures can’t replace quality which comes from quality minded people…people with passion for what they do. Everything trickles down to the consumer, and the only thing they trust anymore, is that they are going to be screwed. This is all about PC and people…AHEM, Management; maneuvering things in such a way so as to avoid consequence for mis-management. They are the ones who draw-up these policies most of the time, and often do so in such a way that implies “cookie-cutter” methods which inhibit creativity! This is an absolute must in technology if you want stay ahead of the curve and the competition. Otherwise, you suffer in key areas of integrity. Client Trial Benefits are a solution to things as I see it. These clients are the one’s I’ve had the most positive experience with, as they enjoy having input, and feel a part of the team…and they are! They are considerate! But when trust and pride cloud the waters of upper-management decision making, things go awry and consumers will see it in the final product. Trust is the core, and the sooner decision makers get their act together, let the good work shine through, it will eventually trickle-down. My advise to you developers- Do what you do best, and mentor the the rest. For what it’s worth: I lost my job clinging to integrity- When my friend returned from a personal client project, he implemented a trust policy in my honor- 30 minute mandatory meetings between tech and management, 1-on-1 every two weeks, technician topic. This forces management to consider the concerns of the technician, and require each to experience things eye-to-eye, making it possible to re-grow trust. Sure it might seem silly to some…the folks stuck in their comfort zone…But isn’t that what caused such a complacency issue in the first place? OOPS Management ISSUE! I would think that developers who fall under management-birthed oppression could benefit from this as well. Sure it’s another meeting…But this meeting actually works if the developer is truly humble in their claim to being passionate for their work…keeping in mind that their is no passion if their is no time to improve one’s character in giving time to people and things outside our own personal comfort zone. The developer’s I collaborated, pushed back escalations before even checking their code…Perhaps they were suffering pride issues, OR they were indeed demoralized by policies and procedures taking up all their time. Time tracking…time stealing…time management…everything but TIME SHARING! Management- Get a clue..Get with your people, and get on with business…True Business Success is founded in trust!

http://www.whatsthebeef.org whatsthebeef

I disagree with the comments on unit testing by more junior developers, for numerous reasons but I am in full agreement of the general point.

There should be flexibility and understanding of developer skill levels, however the management of this is very complex which is one of the reasons why there is a blanket use of processes. If senior developers had more input on process prehaps this could be managed more effectively.

By the way this is a great statement

“if you’re going to say you’re practicing agile development, then practice agile development! A project where you decide before you start a product cycle the features that must be in the product, the ship date, and the assigned resources is a waterfall project”

Suresh

I guess I’m one of the “despised management types” who drank the Agile Kool-Aid after being all Waterfall-y. We have a large dev team, and we do have processes and TDD guidelines in place. However, our devs are NOT measured by the # of unit tests they write or arbitrary test coverage #s. We do plan releases (3 sprints at a time) and define scope, and lock release dates, but the scope is typically re-evaluated based on progress in sprints. For a mature organization, predictability is vital for being credible – if I commit to delivering a release on a certain date, it sure as heck is going out, maybe in a reduced form or with a few known (and signed-off by business) defects.
If any manager is having his team stick to release dates and scope without negotiation, I agree – he needs a brain scan. But don’t generalize!!

However you dress it up, your hypothesis is a return to Wild West days. Do you know of a single developer who would willingly accept to being treated as a second-class citizen and do the grunge TDD work and leave the cool stuff to others?
One more point – In a multi-team environment, how do you expect people to magically know how the other teams are doing on dependencies without a common tool and process to update daily? These are tools to help developers, as much as help management understand where the teams are with regard to a release, and intervene if you see risk.

http://texrat.net Randall "Texrat" Arnold

You raise some thought-provoking points.

I’m wondering if the depth and breadth of process might also be better related to each coder’s past performance. Write robust code? Great– you are rewarded with a streamlined test process. Have a history of inexcusable bugs? Sorry, you still go through a wringer. All subject to ongoing review.

I’m simplifying to keep from writing a novel, but surely this makes some sort of sense…

Eric

In my experience, the only developers arguing against best practices are the worst developers.

Mark

Eric, you must have limited experience.

http://www.revena.com Fabian

That’s why I make some years ago, an objects framework “evenja” that enable only dev creation without process. Enabling to add new features without “thinking” to previous implemented features.

Only now start a project with a big company to show the reality of years of experiences with small local clients (Switzerland).

All object languages enable that paradigm. Developers need to understand the “Quantum Physics”, otherwise… no understanding.
That’s the result of the courses we give to the big company.

The good things is, no more pressure on software project… just simple things, even on big big softwares.

Walt dexter

I have to agree that process in excess is stifling.

I started at me current job nine years ago. Developers, BAs and business users would sit down and hash out what was needed, deva would go get started, BAs would write some requirements, and the code and requirements got reconciled in the end.

That was fun, and it worked, because our developers understand the business. Some have been here for over thirty years.

Now we have BAs who do not understand the business writing requirements that don’t make sense, followed by developers implementing crap. And it’s a lot less fun.

sunil

The first thing they teach you in project management courses is to only apply as much process as is necessary to give your project the best possible chance of success. That means, if you are doing a short microsite, there’s no need for statements of work, executive summaries, project plans etc so those managers apply all this process to your work aren’t doing the right thing, so don’t blame the process, blame the people.

On the software side, I feel if you think writing unit tests is a waste of senior devs or technical architects’ time its probably because they don’t want to take time to understand business requirements. Software is written to solve a business problem not just so a developer can wax lyrical about scala v groovy v clojure.

MichaelJ

I personally think the real issue here has nothing at all to do with developer ability, experience, arrogance, etc. As soon as management has metrics to start, er, managing, any one of these good ideas is doomed. Why? Management 101 teaches us (them?) that “if you can’t measure it, you can’t manage it”. The problem is, the opposite is not true – just because you can measure it, doesn’t mean you can manage it.

Without recognizing that passion, creativity, attitude, and individual problem solving all influence “code quality”, numbers don’t mean anything. Sorry. They don’t. It’s like saying you can manage employee shoe size because you track it in an Excel spreadsheet.

The other problem with blindly tracking numbers is that you will (by default) get the behavior you’re looking for (psych 101). If you tell people what you’re measuring, a group will (in general) alter their short-term behaviors to achieve the desired metric. But in no way does that mean core behaviors have shifted, and everyone magically “gets it”.

Solve the people problems first; the rest will follow.

John

This is why a QA process is essential. I’ve been preaching TDD to my company: Not as a catch-all but to a). Capture the important business logic b). the outliers that are important but often forgotten. c). Iteratively add on more test cases as QA finds more issues.

It should be like a good compiler: Maintain the test-cases so that they’re always very useful (syntax errors, compiler warnings, etc.) but not so much that it gets in the way.

tungstencoil

I mean no disrespect; he does suggest some process is probably good. That’s good. But then he suggests that “better” coders might need less. In practice, that is probably true, but his implication goes farther than “let’s hold the hand of the junior guy a bit more.”

So… how to architects, civil engineers, electrical engineers, graphic designers, technical writers, project managers, and, I dunno… just about every white (and blue, for that matter) collar employee stay “passionate” in the face of some level of process and bureaucracy? How would you like to go on a bridge (or a plane, as others have mentioned) built by folks who are so smart they don’t need any process to validate their genius?

This sounds more like “I’m so good I don’t need to do these dumb things that slow me down” and less “perhaps we should keep the process just at the point it is beneficial”.

I just spent two years with a team of 6 refactoring a critical legacy application built and maintained by people who were “smarter” and “better” than code reviews, TDD, planning, documenting, or hell, breaking that 8K header file into header/class and maybe even into a couple of objects. And truth be told: they were ALL really, really smart…. and really, really willing to cut corners just to “get something done”, resulting in an unmaintainable mess that cost us somewhere in the neighborhood of 1.8MM to clean up.

To all you guys who are WAAAAAY too smart to have processes: that’s awesome. Please just go work somewhere else, preferably someplace I’ll never work.

Jeff J

The general point is an interesting one, but you also sound like a guy who has never had to manage work across multi-team, distributed, complex projects. Without some sane level of coordination and planning, your project is doomed to failure from the start, and subject to another typical developer article – “Why projects fail and why it’s not our fault”.

I also think you are generalizing about the entire industry, based on your limited (and biased) perspective. There is no “right” amount of process (or lack thereof) for every product, company, or group of developers. There is a huge difference in process need between writing a plug-in for Farmville, and writing flight control software – a difference completely discounted in your gross over-simplification.

It’s the difference between artists and engineers – artists hack away on ideas, being “creative”, for the most part doing only what they find interesting and whining about “not having enough tools” or “burdened by process”.

Engineers on the other hand sit back and design a system, analyze the structures, points of interoperability, potential faults, error handling, redundancy, modular self-preservation, upgradability, etc.

If you want to be an artist, knock yourself out – go work at a startup, go do an open source project, read Joel on Software and preach it like gospel – go do whatever you find interesting, building projects of relatively low complexity.

If you want to solve big, complex problems, you need to be an engineer, and understand/accept that solving the problem is a lot more than how much fun you have writing lines of code, or how neat of a code trick you can come up with to show off, and sometimes that requires process and coordination.

But let’s not sit back and pretend that your gross oversimplification (driven by what appears to be your artist approach to programming) is a one-size-fits-all for the entire software industry. That strains your credibility.

http://www.idontgiveafuckaboutyourproblems.com Phil

Am I the only one that would have rather heard the story about pulled pork?

PBarker

The process(es) used should be tailored to specifically address the risks anticipated in the development and deployment of each software project, no more than necessary and no less than sufficient.

Unfortunately, there is almost nothing that humans do worse than objectively assess risks.

Bill

Phil – I am with you. This guy should have stuck with the pulled pork story, because this one reeks of someone whining about how he hates his job.

http://www.robsartin.com Rob Sartin

“Doing full TDD, writing your tests before you wrote any implementing code.”

I think you misunderstand TDD. TDD requires that your write ONE test before your write code to make it pass. It most definitely does NOT require writing ALL of the tests. It would be a Waterfall practice to require all tests. As a software craftsperson, I find TDD enables my creativity rather than stifling.

As for check-in rules, I’d avoid those on small teams. On larger teams, things can get pretty bad if a small number of developers check in bad code. The furthest I ever went was having a CI script that checked a few things (complexity, cyclic dependencies between classes/packages, test coverage) and whined to the developer email list when they got broken.

As for developer participation in planning and understanding user requirements: while I have never done Scrum, I find that having developers actually understand what users want is useful. Certainly you want to balance it with other work, but bottom line is that a developer’s job isn’t writing code it is fulfilling requirements. If the developer doesn’t understand what s/he is building it may be beautiful code, but it will be useless.

As in all practices, balance is critical. I have seen shops with the balance wrong in either direction. That’s why I liked Agile (and signed the Manifesto over seven years ago, after I’d already been doing agile for 4-5 years. The core ideas are great (TDD, test coverage, low complexity) and the problem is turning it into rote rules that must be followed without thought.

Sengan

“Make things as simple as possible, but not simpler” is the solution to most of our programming problems.

Because reading code is often harder than writing it, keeping code simple is key.

For difficult problems that takes hard work and brilliance. For easy but large problems that takes discipline.

Good engineers will use whatever level of process they feel will help them deliver good code. Process is subject to trade-offs like everything else in engineering. For instance, TDD assumes a bottom up development process which is fine for simple tasks, but does not scale to solving very difficult problems which tax one’s limited mental capacity. I find alternating top-down and bottom-up methodologies, followed by writing tests works better in such cases.

Because programmers are not all equal, and much code is inherited, process is imposed on people. This is the cost of collaboration. Splitting projects into independent blocks reduces that cost. Because senior engineers have different process needs to junior engineers, placing them together in the same team working on the same task increases these process costs significantly.

I.e. there is no “one true process”. What works best depends on the task and on your team.

PM Zod

When you can commit to a features+schedule+quality deadline and always deliver, then I might trust you to work without a defined process.

If a developer can’t consistently deliver features of varying complexity to specific deadlines I want them to be following ‘best practice’ to at least enable me to mitigate their under-performance to the standard of quality I, as someone paying them, require.

http://thecodist.com codist

I love all these new things you “have” to do or your code will suck. I lead teams in the 80’s to 90’s which had no scrum, no agile, no TDD, not unit tests, none of this. Yet we delivered applications to hundreds of thousands of people on floppy disks where we had 1 shot to deliver a version that worked (no internet to deliver patches). We worked as a team, communicated all the time face to face or via telephone, tested our own code then had two levels of QA people to validate things. I don’t care what process you do but even in today’s world having a leader who communicates well and is also a developers, smart developers (but not egoists) passionate about the product, and a great QA staff get the job done just as well.

If you think TDD, scrum ,etc work for you then do it. But never assume that what do is the only way.

http://www.nerdwatch.com CptNerd

To the person who commented about building a bridge with no process, do you think engineers design each girder and its relative placement with a previously designed girder, such that at the end they have a whole bridge?

This is the problem I have with a pure bottom-up development process. Without some level of overall design direction (requirements determined beforehand) you’re going to end up with an unstable mess that will pass all its unit tests but won’t do what the end user wants, or will do it so poorly it won’t be worth using.

And “refactoring” is not the magic bullet, either, because requirements can change to the point that whole swaths of code have to be thrown out because the implementation decisions made at the low level and built upon can’t be used even if “refactored”.

Bottom line, Brooks’ statement years ago still holds true, in software development there is no “silver bullet method”, because all projects are different, all customers are different, and all problems are different. What works well with one set of the above will not necessarily work well (or at all) with another set.

Shane

You are implicitly correct. Procedures and overwrought patterns, and strict guidelines, etc are all about stifling creative mindsets. This is a for better or worse type proposition. We all know that without business focus and consistency and bottom line planning, that things would rarely get done and even if they were done, that nobody would buy them anyway. So we need some process and some management, but too much is too much.

I run a small dev shop and have for >10 years. I am frequently admonished for not strictly adhering to design patterns that punish you into becoming an assembly linesman and criticized into oblivion for not writing unit tests until I want to quit. I have never understood how adding more lines of code to a project (i.e. unit tests) somehow saves money. I know somebody will immediately post a rebuttal with evidence that it does, but I simply don’t believe it.

Don’t you then need unit testing for unit tests? Don’t the unit tests need just as much maintenance as the actual code? Don’t unit tests need code review? Doesn’t setting up test data for unit tests take forever? When data models change and propegate themselves up through the code base, doesn’t having unit tests make twice the work? Sure they help accomodate testing of things like column level data change, but every change taking approximately twice the effort as not? I don’t know. You double my workload, and it becomes really difficult to convince me that these processes are saving me time.

I feel as though many of the best practice procedures (which change annually and radically by the way) are suited for particular scenarios, but certainly not all. If something works, I use it. I have worked with more tech acronyms than I care to admit and acquire tens of new ones every year – so I’m not afraid to try something new. It just seems as if every time I try one of the big and stifling patterns, my productivity goes way down. I look at it and say “is this what I want to be doing every day?”

I worked on a project with MS consulting about 10 years ago – long before patterns and TDD were de riguer with MS – and still the project time and dollar budget was laid out like this: 90% proj management and planning, 10% coding. My team did all of the coding over the course of three months and attended exactly three meetings with proj managers. We received almost no input beyond the initial and light spec from proj management. Where did that 90% of the money go???? Poof. It would have been better spent on the blind children for whom the site was being built.

Eric F

I have a friend who is an accountant for the city. When we talk about the processes needed to get our respective jobs done (aka bureaucratic red tape,) he has less. Yup, a government accountant with all of the checks and balances they have in place has less hoops to go through to get his job done.

dude

Whoa @PM Zod… What company? I’ll save us both time and make sure to never interview with you… You just don’t get it, and you’ll have to deal with mediocre devs writing mediocre code until you do.

NervousCat

Let’s just make the process mandatory for the medicocre programmers instead. ;-)

Mike Anderson

TDD is a funny thing. In my opinion it’s just a substitute for good planning. Along the lines of “measure twice, cut once,” writing tests allows you to spell out some examples of how the code ought to be used — from a developer’s point of view — before you write the implementation.

If you want to write out those examples as unit tests, go for it. I prefer just writing out some code snippets in a new TM window. Then I write tests after the fact. It’s almost more like Document Driven Development.

Practically, this means that as soon as your process is more important than your developers, you’ve failed, and aren’t Agile any more. In an Agile world, the point of your process is to make your individual developers work better, and make interactions between developers easier; if process is killing things, it’s gone too far.

Note that this doesn’t mean no process at all; the concept of Agile processes is to capture ways good software engineers work, and formalize them in a way that all software engineers can use to get better results. TDD, for example, works well because it’s one way to get you to understand the problem before you try and solve it.

As soon as you *must* use a process, even when it’s inappropriate, you’ve missed the point; the processes are formalized so that an inexperienced developer can use them without prior experience, not so that you can insist on (say) Scrum for all software. And note that lack of a formal process is in itself a process.

Yes, this is hard; it involves having some competent, experienced developers around to tell whether the junior guys are reasonable when they’re saying “this process is wrong for this bit of project; I’m going to use this other process instead”. It involves understanding enough about what you want to do to know what “working software” means for you. It involves being flexible, and responding when things change underneath you. But it’s what Agile is supposed to be about, not endless process that doesn’t help.

http://www.mempko.com mempko

I am reading many people mentioning TDD as a best practice. Those that think this, why is it a best practice? Is it because there has been lots of real world research showing the benefits? Because you have seen it personally? Or because it is fashionable.

I believe TDD is considered “best practice” because it is fashionable. There simply is not enough research and personal experience demonstrating the benefits to convince me otherwise.

http://danilogurovich.wordpress.com Danilo

You bring up some valid points. I’ve both managed teams and been in them. I’ve been at the strategic level designing the processes that need to intertwine with the rest of corporate processes, and had to endure everyone being pissed because they thought the entire company’s process revolved around their particular niche.

First of all, if you have to shim unit tests to make them work, your code is questionable. I’m fairly libertarian in my viewpoint about TDD, but I use it exclusively for Java and as much as I can with Python and even try to use it with Objective C. It keeps my mind clean and my implementations on the mark.

It also does does something very important — it allows other people to work with your code, and keeps the arrogant rock stars at bay. Being too good to write unit tests will get you kicked to the curb in any group I’ve ever ran — I ask a trap question based on this whenever I hire. I strive for 80% coverage, and then have more tests integrated into Hudson builds. Your mileage may vary, but when you have code that is making a million dollars a day you need to have it be easily maintainable and not have developers be either too afraid or so stupid is to go in to make changes and make things worse.

Process is there for a reason. The process is there not to make sure that your code is reviewed, it’s to build a bond of trust between Business/Development/QA/Release and Executives so they can plan and make good strategic bets. I have been in so many shops where the Engineers think that they work in a software company, but really the software supports the business process of the company, and indeed is often it’s greatest asset. It’s still not a software company. Code bases in this type of an organization (eCommerce!) don’t die. They don’t get “forklifted” successfully either. The base must evolve and it can’t do this without some fairly rigid standards and good documentation. And yes, I write extremely complete Javadocs, Pydocs, etc. because professionals can and should explain why they did things a certain way and how things work, for their colleagues and for those that are still learning. To not do this is much worse

As far as Agile goes, it’s very, very hard to implement Agile in its purest form, but it’s very good when this happens. In any case waterfall is much harder to deal with. Most of the time it is a gray area in-between, often as the requirements of the company are met. Is this bad? Are we being mean to Engineers again?

Who writes the checks writes the rules; being a passionate engineer is great, but lets all remember that we’re not creating art here, no matter how “perfect” we think it is.

Fortunately I don’t have to deal with team management anymore. I’m a senior architect in an R&D group of SomeHugeCompany and we as a small think tank own our own processes — of course if we don’t deliver the goods, the funding goes away.

http://www.jillesvangurp.com Jilles

This is so true :-). We’ve indeed been through some management induced scrum deployment. The results were entirely predictable: it didn’t work. It’s so predictable that any decent scrum training will point this out in the introduction: don’t bother with scrum if it is imposed top down. It’s an interesting disclaimer, given the boat loads of consultancy around scrum.

Process is deadly to creativity. Creativity requires out of the box thinking. Process achieves the exact opposite.

There are no absolute truths in this industry. However a good dose of pragmatism goes a long way to removing the need for process.

Scrum is a double edged sword in the sense that it actually promotes a good dose of pragmatism while simultaneously ‘inspiring’ a lot of people to start involving themselves with non value adding activities and bean counting. As a replacement for more heavy weight process it is mostly good. As a replacement for having no process (which lets be frank, is the reality in most of the industry), it can actually cause people to over do it and get completely bogged down in analysis paralysis and bean counting.

http://agilesolutionspace.blogspot.com/ Keith Ray

“Perhaps an over-the-shoulder code review would be preferable to a formal code review process.”

Yes, we call that “pairing” (except it two people sitting together, not one person looking over another’s shoulder). It saves a lot of time and avoids a lot of waste, compared to after-the-fact code reviews.

ED

It sounds like your exposure to Scrum (and TDD, though I can’t speak to that since I have no experience with it) is based solely on a company who is doing it wrong.

The idea of scrum (at least, the idea used at the company I work for) is that the process should be light weight, and flexible. Have we adopted scrum as out predominant project methodology? Yes. Do we apply it the same to every project? No. Do we use it on every project? No. For us, we see that depending on the work being done, or the team structure, or any number of other variables, scrum may or may not work. In the cases where it does, we let the team decide how the process should be run for their project (I’m guessing they don’t call them self-organizing teams where you work).

As for deciding “before you start a product cycle the features that must be in the product, the ship date, and the assigned resources”, sure doing all of that and trying to force adherence is very waterfall-y. But if you can’t have some idea what you trying to put in the product, and when you might have it done, you shouldn’t be working for a for-profit software company. another bonus to scrum (I think) is that you do this for each sprint. Is there scope creep? Do things change even within the confines of a 2 week sprint? Sure they do. Would I rather have a team of 9 people meet in a 15 minute stand-up meeting, figure out what needs to be done and just do it over having to alter specs, get them re-approved, blah, blah, blah? You better believe it.

Do we have metrics that management looks at? Sure we do, but they are minimal, and used more to start conversations with the team, rather than for blind decision making.

http://nick.com nick

What a bunch of douchey crap. If we follow process of course we’re all doing it “blindly” and not because we fully understand the ‘why’ behind our process. But thanks to developers like you I can demand a high salary to come in and fix the mess you leave behind

Rich

I see a lot of comments along the lines of “but FOO is a very good practice.” I think that they are missing the point of the article.

Lots of things are good practices, but when you make things more about the practice and process than about the output then you throw the baby out with the bathwater. There might be 100 things that would all individually add value, but doing all 100 takes value away.

I think the root cause is that managers and other non-direct-contributors feel a need to do something to be productive, and often this translates into implementing lots of process initiatives. Inevitably these get layered one upon another until we spend 3 hours doing process for every 15 minutes we spend doing something that actually impacts product quality significantly.

Then we buy something more off-the-shelf or 100% outsource because we aren’t cable of even making minor customizations in-house without blowing up the project.

I truly believe that you wrote this article with one motive in mind and was to get a stream of emotional responses which you have accomplished.

Your either trying to do what I just mentioned or your an idiot or a programming dinosaur who has been in the industry so long that he know longer enjoys the art of programming.

I have worked with many talented and extremely experienced engineers over my short 12 year career that had just lost their passion and drive. They started out in the 70’s or 80’s writing complicated C++ programs that was cutting edge or genius at some points. They were considered to be superstars at their companies and sometimes in the industry so therefore they no longer felt the need to increase there knowledge or capabilities.

So when the new, young, passionate developer enters their team, this disinterested, arrogant dinosaur, rebels and resists change.

Working with a team that utilizes any of these best practices makes that team more efficient and effective in building quality applications in a rapid environment. It help them work smarter not harder, but the sanctimonious old timers can not be told different because they learned there best practices during a time when there wasn’t any and that is the canary in the coal mine.

In closing, its rare to see many best practices outside of .NET, Java and Ruby mainly because of one of three reasons.

1) The platform, stack or framework does not allow the flexibility to utilize the aforementioned best practices due to constraints put on the developer during development.

2) There were not best practices in place in the industry when the majority of the engineers were writing code with that language in the beginnings of their career.

3) Lack of standards with a platform or framework.

Jeff

The root cause of this douche baggery is Product Managers, CTOs, CEOs, COO, etc… that do not know any other way other than you come up with a complete product and then set a launch date. Then you work backwards from there.

But that said, Mr Turner is great at writing the obvious but offers no solutions. Hell I can do that! And did!

:)

http://example.com Todd Trimmer

When we did Scrum-like practices, it only entailed one full work day a month and then 10-20 minutes per day for updates. Individuals updated a physical Wall of Wonder. A single individual would then translate everyone’s updates into the software to produce the treasured burndown reports.

I’m also at a loss as to why you insist knowing a ship date ahead of time to be Waterfall. Isn’t timeboxing one of the major cruxes of Agile? We just drop features, not shift the dates.

PLM

You want soul-sucking process? Try developing avionics software to DO-178B Level A. It will make you a better coder but the process just takes all your enthusiasm away. I’ve been doing it for about 30 years and am glad I’ll probably be retiring about the time DO-178C is introduced.

Ruth

@Shane

When data models change and propegate themselves up through the code base, doesn’t having unit tests make twice the work? Sure they help accomodate testing of things like column level data change, but every change taking approximately twice the effort as not?

How else are you planning on testing these changes? By hand? Hoping that you haven’t forgotten an impact or dependency? Missing bugs until they’re live in production? Running tests by hand takes up far more tedious time than running an automated regression suite. You only have to write the tests once, after all.

The Rob

A software process best practice I’ve found incredibly valuable, but I’ve rarely seen implemented:

Don’t discuss software process in writing! Don’t discuss it email. Don’t discuss it in a chat room or IM. Don’t discuss it in an online forum.

Developers can easily spend more time talking about process than actually performing the process.

Jose Luis Soria

It’s sad to see that most people criticizing Scrum, in fact haven’t had the opportunity to practice (or even learn) it properly.

“Tracking your time to generate burn-down charts for management.”
“maybe junior (or specialized) developers should be writing the unit tests”
“micro-manage them with daily updates to VersionOne to make sure they’re going to make their sprint commitments”
“morass of meetings and metrics-gaming”

James, statements like these demonstrate that you have a distorted view about how a Scrum team works. As George stated in the first response, maybe you haven’t had the opportunity to work in a proper, non-dysfunctional Scrum environment.

Jose Luis Soria

Sorry about the multiple comments above… I hit the button more times than intended ;-)

http://chiggsy.com Kevin

We see here from these comments that this is religion, not best practice

Marc Solomon

I have worked in the ‘wild west’ arena and I have worked for CMM level 5 companies. Passionate programmers will create great code no matter where they work. Poor programmers will create crap code no matter where they work. Process will catch much of the crap poor programmers create, but not all. Worse, many poor programmers will find ways around the process just to meet deadlines.

One of the main flaws to your logic is that process kills passion. Poor process, poorly implemented process, overbearing process will all kill passion. Good process, though, will assist passionate programmers.

The problem being that getting to ‘good process’ is a hit and miss process. What is ‘good’ process for business applications is frequently ‘crappy’ process for realtime applications. Process needs to be tailored for the application, the workplace culture, and the management goals. While there are many ‘best practices’ in the arena of process, I have seen no panacea for the ultimate process.

How do we maintain the passion in the passionate programmers while we look for the process that will reduce the harm done by mediocre programmers?

When the gurus of process brought the message of process to me, I accepted it kicking and screaming the whole way. What turned me around was when the responsibility for improving the process was thrust upon me.

For many programmers, process is something being forced upon us. When it is presented to us as ours to control and enhance, when we take responsibility not only for the code but the process that is used to create it, we can than start to show passion for the process as well as for our code. Stop shoving process down developers throats and start letting the developers develop the process they will use (and own)!

Rui

Well, this sentence “if you’re going to say you’re practicing agile development, then practice agile development! A project where you decide before you start a product cycle the features that must be in the product, the ship date, and the assigned resources is a waterfall project.” is my exact opinion. You can’t be agile with all this rigid points that define everything.

Processes are great because each has their own advantages and can give significant improvements, but each case is a different case and I have to say I personally don’t believe in following blindly an entire process. I do believe in following principles derived from processes which, perhaps with some modification, allow to extract maximum productivity from human resources. That’s what I truly believe is “agile”.

Mark

Honestly, I can say what kills MY developer passion is having to maintain projects where someone decided they were too clever by half and didn’t adhere to some sort of standard, or best practices.

I’ve work on wild west projects, and I’ve worked on DO-254 projects (hardware variant of the DO-178 standard mentioned above.) Process has a place based on how mission critical your design has to be (i.e. how many people are going to die if someone screws up.)

Generally speaking, I’d rather see SOME process than none. And highly rigorous process should go to the highly rigorous designs. The rest is a judgement call.

Max

What nobody yet commented on is what percentage of people who should even be programming at all.

There are few gurus, a moderate number of really adept, a big middle zone of average that are careless at times, and a small percentage of unruly who do astoundingly odd things like try to shift blame, test one happy day scenario and declare they are done, and so forth.

In such a mixed crowd, the bigger companies simply can’t let the small unruly or sometimes careless destroy their releases, and end up being forced to create restrictive rules.

Of course the gurus and adept go crazy in such an environment of excessively rigid rules and fascination with metrics. With any luck, they will move up and out and the big middle zone of average probably increases making it worse.

http://nicoletedesco.wordpress.com/ Nicole C. Tedesco

Software process, like “method” in science, is bunk! I finally understood what the philosopher Paul Feyerabend was trying to say. What is important is the data, or the stuff of reality, otherwise known as results. Experiment (software tests), deduction and induction are all very important but no two people are going to arrive at their conclusions in the same way. That is, no two people process data and leverage their capacity for deduction and induction in the same way. One person’s process (method) is another person’s confusion. If you want to de-motivate a creative scientist or software developer, force them to think like someone else who isn’t them.

Process is no substitute for knowledge.

Tyler

If you have a team where the management (executive, marketing, product, engineering) can clearly produce a specification of features that requires no change, and developers that can estimate EXACTLY how long the solution (i.e line, block, class, library, component, system, complete application) takes to produce without failing to hit the timeline, AGILE would not be a development team process, but simply a concept used to help prioritize work.

http://www.youtube.com/watch?v=QH2-TGUlwu4 BugsCostMoney

Couldn’t disagree more with the thoughts about “seasoned” developers not having to write unit tests for a few reasons.

First off, “seasoned” is subjective.

If we go by title as a measure of seasoning, companies don’t promote by programming skill alone. There’s politics, who knows who, who rode coattails, and darnit we just have to promote whomever because someone left and a director wants to keep that position underneath them rather than lose an employee slot.

If we go by number of years programming, that means nothing either. I’ve met developers who can code circles around RPG 40+ years of industry experience, and visa versa.

If we go by # of certifications, we’re getting warmer. But that only measures the capability to code well – not the desire to code well vs take shortcuts.

And that’s what this all comes back to. Desire to do what’s right. Taking shortcuts in the wild wild west with will smith, versus testing your code before it’s in production & having confidence that stuff won’t break in production.

Because it’s far better to know it’s broken BEFORE it’s deployed than AFTER it’s deployed. Why? It costs less.

I’ve personally seen lack of testing screw up 10,000+ orders over a few days. Someone makes a change and ninjas that change up into production, no unit test, no testing. That’s really unfortunate, because that’s not only lost revenue, but costs in bug fixing, deploying, meetings, conversations, customer service taking phone calls from angry customers, those customers never purchasing again, bad reviews online, and so on.

Compare that versus some ran a unit test on their local development environment, and said “Oh. Looks like this is wrong. Here we go, one line fix, and now I get a green bar. Good thing I found that before it went to production!”

http://techdoertimes.com Sergio

Interesting post, thanks for that.

I can understand the notion of an Agile practice with a narrow focus on boosting the productivity/quality of an individual’s work

We are all different, and there’s no telling that practice X will be as effective for for John as it is for Mary.

However, there are many Agile practices that are focused on boosting not an just the quality and productivity of an individual’s work, but also that of their team, and organization. These practices whose scope extends past the individual are necessary to help develop the competence of colleagues, and their ability to be effective for the organization.

Can you really say that you would be demotivated If you need to be a bit more inefficient in order to benefit the effectiveness of your team?

http://brainslink.com/ Vin D'Amico

Achieving the right level of process is hard. Many companies introduce too much process as a way to get the level of command-and-control that they want. Of course, too much process merely wraps waterfall development in an agile wrapper, as the author points out.

If you find yourself in a too-much-process situation, see if management will agree to relax one aspect of the process and evaluate the results. Then, try relaxing another aspect, and another. The key is to show good results even as the process is relaxed.

You may never achieve a truly agile approach but you may get to a situation you can live with.

http://kfsone.wordpress.com/ Oliver "kfs1" Smith

It’s far more commonly the toolset than the process.

We can all see the irony of a coder, who insists on using Notepad and Explorer as his IDE, bemoaning the time it takes him to manage unit tests and documentation…

Far too many teams have one or more programmers throw together an assortment of open source apps with a mix of randomly-languaged scripts to create some illusion of interoperation between them.

And most process-sucks programmers I’ve come across /insist/ on using their personal selection of tools that do nothing, or even hinder, good development practices.

Personally, I’ve turned the tables; process doesn’t own me. When I approach a professional assignment – and lately even personal hobby projects – I *insist* on various levels of process.

When I sit down and start coding, I want a certain degree of automated QA provided for me. Documentation and Unit Tests that are generated without slowing me down.

See – if I’m being handed yet another save-the-world project that needs the green-glowing gray matter that earns my keep, I don’t need to be worrying about checking for null pointers and crap.

It’s 2011, why the bloody hell do we not have editors and a language that can facilitate all the good practices for you so that we programmers can get on with the task of telling the thing what we want done?

http://ar3.me Andrew L. Robinson III

Bottom line…process for the sake of process is a scar.

In software, it is about freedom, creativity and self-motivation.

If you hire developers that don’t have this, or you have management, that wants to “manage” a project, instead of set great context and get out of the way, you will end up with a shitty end result regardless of what you do.

The Agile Manifesto is great, but use the concepts when it makes sense.

TDD can be great, but not as a fucking mandate.

Not trusting your developers will ultimately lead to an incredibly shitty drone-centric environment, but you’ll be spending a shit ton more money to get to this level of shittiness…

Hire the right people, set context for them, use common sense and get out of the way!

Avinash G A

I prefer coding first, and then writing the test case for only those methods which i am not 100 % confident of.

Writing a test case for every public method (getInstance() (. .)) is real shit.

http://agilecraft.wordpress.com/ Petri Heiramo

There’s been so many comments already on the “Agility” of the managers in the example (i.e. process over people), on “seasoned people not needing to write unit tests” and “juniors doing the tests”, that I’ll skip that for now. I think much of the latter is just exasperation on the part of the author.

I’ll focus on the use of “best practice” that has been mentioned in this comment thread many times.

Best practice doesn’t exist.

Or maybe it does, but not in Complex space. Only when things are predictable can we say that some approach is better than any other, in all circumstances. However, as soon as we get emergent behavior and too many factors to count, such certainty of a single best way disappears.

Instead, we have a group of good practices, and even that group is open-ended (i.e. we continually learn new good practices, some of which may be highly situational to a certain situation). It is the challenge of the development team to know enough of them to make a good choice, decide how to best apply them into current situation, and continually tune the use to improve their ability to deliver working code.

That’s pretty easy to understand and implement in a single team. When there are more teams, it’s not the job of the management to make the decisions; it’s the job of the development teams themselves to discuss the good practices, organizational needs and application, and ensure consistent _sensible_ use in all teams. Imposing any single process on everyone is a mistake in both circumstances.

In the author’s case, the key problem, I think, lies in the management. Which is not surprising, as I think it is the biggest problem in software development these days. :)

pc

to Petri,

this is always a problem, not only in software development – if you introduce processes (which may be construed as being good) that means that you are automatically introducing a managerial layer which has the authority to implement these processes, and you might kill the company in the process

laughingskeptic

I think a key point about whether process kills developer passion is the attitude of the organization towards process. If process is dictated top-down and adherence is demanded regardless of when it makes sense, then developer passion is killed and the process comes to be seen as inteference. On the other hand, if the instution’s attitude toward process is that it is there to help the developers to be the best that they can be, then processes can actually increase developer passion. In the latter environment you will find that across the board that stress is also much lower. I have been working in software development since 1981 and have seen the entire spectrum between these attitudes at many different organizations and the ships that run smoothest follow the latter course.

Shane

@Ruth

Ruth, you skipped over the part where you have to recode the unit tests to accommodate change and test the unit tests. How is coding extra code to prevent errors in code not a problem? How does it cause less work to happen? My point is this: making changes in code sucks and requires lots of different testing. Forcing code-based unit testing in every scenario because a bureaucrat says so means little to me. It isn’t as clear cut as any of this.

Many have said it, and I agree. Good coders tend to produce good code. Crappy ones produce crappy code, including crappy unit tests, crappy adherence to processes which begets even crappier code.

***********
You unit test, by hand. You QA, by hand. That testing has to be done anyway. Robots can’t put a stamp of approval on a product; you can’t “ship it” based on their go ahead. Strong typing eliminates much of the issue in the regular code when there’s that data change anyway.

When data models change and propegate themselves up through the code base, doesn’t having unit tests make twice the work? Sure they help accomodate testing of things like column level data change, but every change taking approximately twice the effort as not?

How else are you planning on testing these changes? By hand? Hoping that you haven’t forgotten an impact or dependency? Missing bugs until they’re live in production? Running tests by hand takes up far more tedious time than running an automated regression suite. You only have to write the tests once, after all.

Interesting perspective. I personally think if Scrum is done right, then it will expose some of the problems which are normally hidden under the rug of traditional long development process like waterfall. And we are now seeing them in bright daylight and we need to deal with them. The XP practices are the powerful tools to address some of them. I have just started a series of blogs on this topic, see http://www.agileiq.org/2011/05/16/how-should-we-deal-with-the-mess-that-scrum-exposes/

“the test cases took two- or three-times as much time to code as the actual code, or where having to shoehorn in shims to make unit tests work has reduced the readability of the code”
These are symptoms of people are not actually doing TDD. Each unit test should be written in

It’s less the following of ‘best practices’ and more the following of any practice that either doesn’t fit or doesn’t make sense to those involved. Either help the people involved understand the purpose of the process, customize the process down to just the parts that provide value, or do both.

TDD does not mean you have to write all your unit tests beforehand. That does not make any sense. That defies the whole concept of being agile. Writing tests first is just one aspect of TDD, TDD also ensures you’re writing testable code. If your code is not unit testable, you better refactor it to make it testable. With that said, I agree with you that developers are spending more and more time on the process, other than writing code, however, I wouldn’t include TDD as part of the process.

David Edwards

The amount of time is takes to perform a task is inversly proportional to the amount of times your asked how long it will take to be done.

Mike

Most of my problems happened when I solved the issue out of the overall context of the project. Many times a problem becomes less difficult as other priorities solve it for you. Don’t ignore the fact but keep it close while working on others. It is hard to do well when all the factors are still being discovered.

In a team discipline keep open to many changing influences. As in life, my problem is the hardest and everyone’s are easier. Listen, don’t retreat, offer your thoughts when it is wanted, be professional. The big play might get you the wink but how about the future teammates? The scrum at the end has its place just like the start. It is your responsibility to manage your confidence.

Bruce Rennie

Personally, I disagree with the authors definition of “passionate”.

I’ll be honest, I’m a little weary of the “whoa is me” programmer mentality that proposes that they are victims of process or management or whatever.

You know (or should know) that the anecdote “I’ve worked on some projects where the test cases took two- or three-times as much time to code as the actual code” is probably irrelevant. It’s not the time it took to write the tests that matter. It’s the time you’re going to save over the life of the code base that is the point. If you don’t believe the investment in the tests is going to pay off then say so.

If a programmer feels that the process doesn’t fit the context, then say so. If a programmer feels that “shoehorn(ing) in shims to make unit tests work has reduced the readability of the code”, then say so. Or find a better way. Or do something, anything. Make a case. Put forward an argument. Be “passionate” about the project.

But for God’s sake, stop looking at the profession as simply the act of banging out as much code as humanly possible in an 8 hour period.

Featured Video

The Internet of Things That Do What You Tell Them: Cory Doctorow passionately explains how computers are already entwined in our lives, which means laws that support lock-in are much more than inconveniences.