Friday, June 06, 2014

TL:DR You can not observe a developer without altering their behavior.

First a story.

Several years ago I worked on a largish project as one of a team of developers. We were building an internal system to support an existing business process. Initially things went very well. The user requirements were reasonably well defined and we worked effectively iterating on the backlog. We were mostly left to our own devices. We had a non-technical business owner and a number of potential users who gave us broad objectives, and who tested features as they became available. When we felt that piece needed refactoring, we spent the time to do it. When a pain point appeared in the software we changed the design to remove it. We didn’t have to ask permission to do any of things, so long features appeared at reasonable intervals, everyone was happy.

Then came that requirement. The one where you try to replace an expert user’s years of experience and intuition with software. What started out as a vague and wooly requirement, soon became a monster as we started to dig into it. We tried to push back against it, or at least get it scheduled for a later version of the software to be delivered at some unspecified time in future. But no, the business was insistent, they wanted it in the next version. A very clever colleague thought the problem could be solved with a custom DSL that would allow the users themselves to encode their business rules and he and another guy set to work building it. Several months later, he was still working on it. The business was frustrated by the lack of progress and the vaguely hoped for project delivery dates began to slip. It was all a bit of a mess.

The boss looked at this and decided that we were loose cannons and the ship needed tightening up. He hired a project manager with an excellent CV and a reputation for getting wayward software projects under control. He introduced us to ‘Jira’, a word that strikes fear into the soul of a developer. Now, rather than taking a high level requirement and simply delivering it at some point in the future, we would break the feature into finely grained tasks, estimate each of the tasks, then break the tasks into finer grained tasks if the estimate was more than a day’s work. Every two weeks we would have a day long planning meeting where these tasks were defined. We then spent the next 8 days working on the tasks and updating Jira with how long each one took. Our project manager would be displeased when tasks took longer than the estimate and would immediately assign one of the other team members to work with the original developer to hurry it along. We soon learned to add plenty of contingency to our estimates. We were delivery focused. Any request to refactor the software was met with disapproval, and our time was too finely managed to allow us refactor ‘under the radar’.

Then a strange thing started to happen. Everything slowed.

Of course we had no way to prove it because there was no data from ‘pre-PM’ to compare to ‘post-PM’, but there was a noticeable downward notch in the speed at which features were delivered. With his calculations showing that the project’s delivery date was slipping, our PM did the obvious thing and started hiring more developers, I think they were mostly people he’d worked with before. We, the existing team had very little say in who was hired, and it did seem that there was something of a cultural gap between us and the new guys. Whenever there was any debate about refactoring the code, or backing out of a problematic feature, the new guys would argue against it, saying it was ‘ivory tower’, and not delivering features. The PM would veto the work and side with the new guys.

We became somewhat de-motivated. After loosing an argument about how things should be done more than a few times, you start to have a pretty clear choice: knuckle down, don’t argue and get paid, or leave. Our best developer, the DSL guy, did leave, and the ones of us arguing for good design lost one of our main champions. I learnt to inflate my estimates, do what I was told to do, and to keep my imagination and creativity for my evening and weekend projects. I found it odd that few of my new colleagues seemed to actually enjoy software development, the talk in our office was now more about cars than programming languages. They actually seemed to like the finely grained management. As one explained to me, “you take the next item off the list, do the work, check it in, and you don’t have to worry about it.” It relieved them of the responsibility to make difficult decisions, or take a strategic view.

The project was not a happy one. Features took longer and longer to be delivered. There always seemed to be a mounting number of bugs, few of which seemed to get fixed, even as the team grew. The business spent more and more money for fewer and fewer benefits.

Why did it all go so wrong?

Finely grained management of software developers is compelling to a business. Any organization craves control. We want to know what we are getting in return for those expensive developer salaries. We want to be able to accurately estimate the time taken to deliver a system in order to do an effective cost-benefit analysis and to give the business an accurate forecast of delivery. There’s also the hope that by building an accurate database of estimates verses actual effort, we can fine tune our estimation, and by analysis find efficiencies in the software development process.

The problem with this approach is that it fundamentally misunderstands the nature of software development. That it is a creative and experimental process. Software development is a complex system of multiple poorly understood feedback loops and interactions. It is an organic process of trial and error, false starts, experiments and monumental cock-ups. Numerous studies have shown that effective creative work is best done by motivated autonomous experts. As developers we need to be free to try things out, see how they evolve, back away from bad decisions, maybe try several different things before we find one that works. We don’t have hard numbers for why we want to try this or that, or why we want to stop in the middle of this task and throw away everything we’ve done. We can’t really justify all our decisions, many them are hunches, many of them are wrong.

If you ask me how long a feature is going to take, my honest answer is that I really have no idea. I may have a ball-park idea, but there’s a long-tail of lower-probability possibilities, that mean that I could easily be out by a factor of 10. What about the feature itself? Is it really such a good idea? I’m not just the implementer of this software, I’m a stake holder too. What if there’s a better way to address this business requirement? What if we discover a better way half way through the estimated time? What if I suddenly stumble on a technology or a technique that could make a big difference to the business? What if it’s not on the road map?

As soon as you ask a developer to tell you exactly what he’s going to do over the next 8 days (or worse weeks or months), you kill much of the creativity and serendipity. You may say that he is free to change the estimates or the tasks at any time, but he will still feel that he has to at least justify the changes. The more finely grained the tasks, the more you kill autonomy and creativity. No matter how much you say it doesn’t matter if he doesn’t meet his estimates, he’ll still feel bad about it. His response to being asked for estimates is twofold: first, he will learn to put in large contingencies, just in case one of those rabbit-holes crosses his path; second, he will look for the quick fix, the hack that just gets the job done. Damn technical debt, that’s for the next poor soul to deal with, I must meet my estimate. Good developers are used to doing necessary, but hard to justify work ‘under the radar’, they effectively lie to management about what they are really doing, but finely grained management makes it hard to steal the time in which to do it.

To be clear, I’m not speaking for everyone here. Not all developers dislike micromanagement. Some are more attracted to the paycheck than the art. For them, micromanagement can be very attractive. So long as you know how to work the system you can happily submit inflated estimates, just do what you’re told, and check in the feature. If users are unhappy and the system is buggy and late, you are not to blame, you just did what you were told.

Finely grained management is a recipe for ‘talent evaporation’. The people who live and breathe software will leave – they usually have few problems getting jobs elsewhere. The people who don’t like to take decisions and need an excuse, will stay. You will find yourself with a compliant team that meekly carries out your instructions, doesn’t argue about the utility of features, fills in Jira correctly, meets their estimates, and produces very poor quality software.

So how should one manage developers?

Simple: give them autonomy. It seems like a panacea, but finely grained management is poisonous for software development. It’s far better to give high level goals and allow your developers to meet them as they see fit. Sometimes they will fail; you need to build in contingency for this. But don’t react to failure by putting in more process and control. Work on building a great team that you can trust and that can contribute to success rather than employing rooms full of passive code monkeys.

88 comments:

Wow - I recently wrote a post about building teams based on all the experiences and observations you list in this post: http://www.ntcoding.blogspot.co.uk/2014/05/a-strategy-for-building-software-team.html.

True on so many levels. Reminds me of the anecdote about programmers as people whose lives depend on getting eight consecutive hours of sleep, yet managers tend to sit on the edge of the bed asking "are you asleep?" every hour on the hour.

"If you ask me how long a feature is going to take..."That para is pure gold.

However, I'd like to point out one important positive that Jira brings - PM tools like Jira, i feel, are used to ensure that a Minimum work output is there from developers. Over-reliance on them can cause all the problems you mentioned in your post.

I agree completely and have experienced exactly this, multiple times.But, I have a question. How did "That Requiremeng" get resolved? Did it get built? The autonomous developers seemed to be struggling with it. Did the micromanaged developers get anywhere with it?

I mostly agree with your arguments but I would like to add something about it. Most of the problems is that PM don't understand how software development works. In my company we use agile metodology and I find it useful ONLY if PM and clients understand how we develop software. It's useful for us (developers) to plan and slice in smaller "problems" and focus in them.Although we use 2-3 week sprints and we have a meeting every morning to see if there were problems or we need to adjust or whatever. A lot of times we underestimated user stories. Some others we overestimated. We usually estimate on difficulty more than in time. Also our PM DO understand the need of refactoring, the need of making good code so we really focus in the quality of the product and not only on delivering stuff.

I'm happy where I work because of that. Before being in this company, I was in others just like you describe. At the end you get demotivated developers or other people that are not really creative but only want to get something done, without much thinking, deliver it and go on. I think you can't do good quality software like that but unfortunately that's what a LOT of software companies are. And PMs obsessed with points, delivery in time, and such are mostly on those companies.

Sure Ollie, that is how it works. If employers want passive code monkeys they'll get exactly that, and more bugs and features that take longer to create, etc. His point is that employers would be wise to let go of their need to measure everything and get more motivated emplyees that do a better job. If employers don't like that they can ignore the advice and carry on business-as-usual.

If employers want passive code monkeys they'll get exactly that, and more bugs and features that take longer to create, etc.

Employers don't care about developers, they care about making money from software, if they are making money from 10,000 lines of spaghetti code they won't care about anything else. If they aren't making money they'll probably go under, at best they might try again with better people.

Developers don't seem to realize you are just a commodity, you're hired for your experience & intellect and employers are very well aware at what level they are recruiting.

Autonomy isn't a panacea either. In my time I've seen it allow some really bad examples of cathedral-building and CV-driven-development.

The scenario you described was certainly dysfunctional and a classic example of how to destroy a culture, but it sounds more like the root of it all was a failure of the organisation to understand the real needs of their real end users, and that this (management) mistake getting compounded into the destruction of a high-functioning dev team.

I am 100% for autonomy, but not if the team is autonomously fixing the wrong problems...

I'm on Ollies side here. It's not that I think employers intentionally set out to make lives miserable for the autonomous developers - they just have different interests. They need to run a business and sell their product. So when they ask developers: "How long is it going to take", what they are really asking is: "How long do I need to go on paying you before I see some return on my investment that I can get customers to pay for?".

If you want to be allowed to work autonomously you need to provide some transparency for the management. One way is to be specific about difficulties, fx. explain that features are not well understood or that requirements are too vague. Be specific about what is not understood or what is too vague. Eric Evans calls this creating the ubiquitous language between business and developers.

This seems wonderful and makes me envious of those that get to work on internally funded projects! At least in that situation you have the potential to demonstrate to your employer that it works.

I dare say you wouldn't win much work though turning up to pitch to potential customers and declaring that you'll definitely deliver something, but you don't know how long it will take or how much it will cost.

Maybe we should be trying to convince the customer that this way works, and that choosing a developer should be more like commissioning a painting than building an extension to your house.

I guess this would be done by demonstrating a portfolio of previous projects that weren't done to a fixed cost but were successful.

I suppose one advantage we have over the artist is we can offer something like "Pay us some fraction of the budget and make your decision on whether to continue with us based on what we deliver in that time". Although this would be unlikely to work when there's a fixed timescale.

An interesting article ... mainly it's an interesting illustration of how to blame the tool, rather than the wielder.

These days,I've moved on from writing code, to effectively being the project manager for three separate projects. Having been on both sides of the fence (on more than one occasion), here are my thoughts on this article:

1. Jira is a really good piece of software. Just as someone can write shit code with C#, so someone can do shit things with Jira. Used well, it's an incredible tool.2. The PM you describe is not a good one. A good PM of a software project will know what technical debt is and will know that time must be spent writing tests, refactoring etc. He will know that some tasks will massively overrun. He will know how to balance the need to meet deadlines versus the need to create quality code. In shot, a good PM will empathise with developers.3. Like many things, it's all about balance. Autonomy is good. So is accountability and transparency. Let developers plan a sprint and estimate tasks, then leave them to it until the next planning session (save for daily standups to let them report problems and successes).4. Find a balance between leaving a developer to work on one task for months and micro-managing her day. Both extremes are crappy. The middle ground works well.5. If you find yourself increasing your estimates, due to the previous ones being too low, how can this not be a good thing? Far too many developers (me included, all too often) under-estimate tasks.

One successfully manages developers like one manages everyone else: by seeking the best balance between autonomy and process. The exact best balance will vary between teams. What's certain though is that too much autonomy and too much process are both very bad things.

I think this article describes a very stereotypical set of scenarios, whith PM's and developers being portrayed as caricatures of the real thing.

Ive been a freelance enterprise developer for 15 years, and there are pro's and con's to micro-managed teams as well as unmanaged teams.

Ive worked in just about every possible team configuration and there is no sure-fire recipe for success. If you not able to be ingenious, imaginative and capable - your arent a great developer. If you cant floursh in a micro-managed environment - you are not a good developer.

A good developer understands that team-based software development is very complex, with many different convoluted paths to success.

I dare say you wouldn't win much work though turning up to pitch to potential customers and declaring that you'll definitely deliver something, but you don't know how long it will take or how much it will cost.

A sad but typical story. Unfortunately the process management types think they provide consistency because the collect data about developer behavior, and through the use of statistics, they can normalize that behavior and bring "control".

The key element that they miss is that these missions that then send developers on are NEVER THE SAME, so it's like asking a football team to describe how the plays will proceed. They can never tell you, because the ball bounces weird, someone twists an ankle. etc. Software development utilizes methods, but not process in the same way manufacturing does.

"it's like asking a football team to describe how the plays will proceed."

That is exactly what software development is like. The difference between good developers and bad developers is that a good developers will have an offensive/defensive formation and player responsibility given out. Bad developers are like sandlot teams that just line up and all do dumb things.

Good developers will give you a fighting chance while the bad developers may get a bounce here and there, but eventually it will come to and end.

I chose to go down the Scrum route because it positively encourages developer autonomy, but with an eye on what the business thinks is the next high priority item. Devs choose the solution, Business chooses the next item. As Scrum Master I must ensure that devs are given the best environment in which to flourish. If estimates are killing creativity, then my job is to remove that burden.So.. project management per se is not going to deflate devs; but bad project management will.

That post rings very true. The one thing I'd say is that Jira and other project management tools are very helpful of keeping track of things you want to do in the future (otherwise, those things get lost). The problem comes when someone wants "predictable execution". That is where the downward spiral of punishment and padding begins. Predictable execution is a recipe for mediocrity and a predictably glacial pace of development. My personal rule, "no points!". Whenever I've been required to give things points... just give them all the same number. On average, it works out.

The company that I work for recently went through a restructure. All aspects of Dev., whether they be production or in-house support systems are now managed like they were Sales or some other department where "how long is this going to take?" is an easy and concrete question to answer.That's one of the many reasons I'm the only Dev. left and also why I'm leaving my job shortly for a better opportunity. The moments I was left alone and was able to wow management based on their lack of expectations and when I came up with solutions to problems before they were problems were the best of times. Being micromanaged and forced to kick out shit software on an unreasonable time schedule is not only career killing, but soul stealing. I measure my career worth in inverse proportion to bug fixes.Thanks for sharing.

I get the impression that the only people who replied to this, and loved it, are developers. While I understand the desire and even the need for autonomy in order to harness creativity, please keep in mind that the owners of the business have a responsibility to keep the business in the black. It's childish to assume that you can just let everyone do what they want on their own schedule. In time, it will fall apart. Finding the right balance is critical and something all good owners/managers will do.

Good article. It and the subsequent dissenting comments just show that there is not a single approach to software development but a range of options based on the circumstances.

The author made it clear that their approach to interpreting the user's requirements worked very well and the users (clients/customers/business) was very happy until they hit a requirement that was too vague and really needed a subject matter expert to better define. In my mind, this is where the fine-grained approach for this specific requirement could have helped.

So, I think it is better to be open to different development approaches depending on the situation. I do feel that the software-engineering/fine-grained approach should be the exception and not the rule as it is often overkill.

IMHO a good PM should "manage" client expectations, the quality & depth of requirements statements, scope creep etc, but really, shouldn't be aiming for any level of developer management or control - the best PM's are those who _enable_ project developers - ensuring they remove blocks and enabling the developer to concentrate and build the best code they can that fulfills the requirements. End of.

Great article. It explains the problems caused by micromanagement perfectly. I would add that not only do companies lose good developers but also cause good developers not to apply for their job openings. I refuse to consider any job that requires Agile, Scrum, daily standup meetings, pair programming, Xtreme Programming, and so on. It is all micromanagement!

A good software developer or team leader knows what to do and when to do it (when a meeting is required, clarification is needed, communication is required, and so on). Also, estimates are part of the business. I don't have a problem with estimates. However, a developer should be given a little time to deliver a meaningful estimate (which may require some experimentation and research).

Hiring an experienced architect would be my first recommendation. Break teams into smaller groups of max 5-6 devs., architect defines coding standard, creates architecture and performs code reviews. PM takes resource allocation directives from the architect. Architect isoloates areas of volatility, and deals with changes. Senior (and team leader) devs consult with architect regularly. PM works with architect on schedule, integration path etc. Being a good architect is not just knowing the architecture, it is about "driving the machine". Hiring devs in the middle of the project to speed up the progress is suicidal as is giving PM anything else besides a Gantt chart...no amount of SCRUM or Agile will fix broken architecture integrity. SCRUM is stupid and does not work. Agile is fine but has to be implemented correctly. Anything that starts with re-, is bad; refactoring, redoing, rebuilding...and then re-testing. Thank you for the story...classic example of "software crisis" see Wikipedia on that. Cheers!!!

brilliant post!I mostly agree with your arguments but I would like to add something about it. Most of the problems is that PM don't understand how software development works.If employers want passive code monkeys they'll get exactly that, and more bugs and features that take longer to create, etc. http://tricityjunction.com

3. By all means, blame estimations measured in hours. For software, the only estimation worth anything is relative estimation, and the intention of that is primarily to help with planning/backlog refinement.

4. Blame the untrained PM who is trying to implement a practice no-one understands.

Solution: If you use SCRUM, train the whole team in all of the roles. That way everyone knows the rules and why the roles are important. If management says no, get a new job. Future projects are doomed due to scatter gun mismanagement of a bit bucket of issues using process for the sake of process.

Speaking of technical debt, there's a wonderful tool called sonarqube - it shows technical debt in terms bosses can understand, like number of working hours, it also paints criticals and majors in red colors etc etc, in a word, wonderful!So anyone in need of explaining code quality to bosses, hook up your code to sonarqube, makes it waaaay easier.

My SCRUM project manager stood up for developers.And it's supposed to be like that. According to methodology, project owner is supposed to press issues from customer's point of view, while project manager advocates developers view.

Thank you so much for this post. You absolutely capture the loss of joy in my work, and the soul-numbing apathy I feel toward my day job development assignments, since my workplace has implemented a granular command and control structure.

For some, an additional casualty could well be a sense of confidence in one's abilities. Implicit in micro-management is the message that you are either incompetent or untrustworthy (or both). Hear that enough and you might start to believe it.

My two bright rays of hope are that I will get a new day job, and luckily, I am able to do some side projects, so I can remember how it feels to be a trusted, professional developer.

Agree with your comments at the end developers are person that need to be free to make decisions, but I think agile methodologies like XP allows to create great teams because all the members can say their opinions. I think examples like you wrote the problem is a bad management not the methodology. BTW, great post.

Micromanagement denotes multiple things:- Need to sync frequently = sign that people is not on the same page. Compensates for lack of planning, estimation and communication skills.- Need to verify progress frequently = lack of trust. Compensates for people that are not honest about being blocked.

If you want to end this situation: update your tickets with details about your progress. If someone asks you for an update, refer them to the ticket. If they have a question, refer them to the ticket. Keep all the project management discussions on the issue tracker.

Then, compare, what is better:- fixing a problem with time (preventive) or fixing a problem with no time, customer and reputation impact (reactive).- slower at first but sustained development speed and scalable team growth (controlled technical debt) or faster at first but decaying development speed with developer/bottlenecks (uncontrolled technical debt).- to invest in your employees and lead on consensus, or keep an army of demoralized people and lead on fear and authoritarism.

I wrote an article about a year ago called Give developers autonomy which agrees with your conclusion. Bear in mind that autonomy is not a good motivator for all types of work. My article has a TED Talk video from Dan Pink which covers this.

I totally and whole-heartedly concur with this article. This is exactly the challenge I face. That said, I also understand the challenge that product management, or sales, or marketing have. They can't loiter around waiting for a whisper of a promise.

For those who want to ship half an airplane half of which has been designed on half a piece of paper, good luck with the half-assed work. Fire the guy who asked for thehalf-assed work to be delivered to the passenger when your stock tumbles, notthe guy who told you about problems you could have prevented in the first place!Half of good luck.

"A sad but typical story. Unfortunately the process management types think they provide consistency because the collect data about developer behavior, and through the use of statistics, they can normalize that behavior and bring "control". "

Exactly this. I also had a situation once where everything was going great, devs were happy to put in overtime because it felt like our project. We kicked ass everything worked, the company succeeded so the owners (non IT) decided to get managers to lock in the success, and then hell broke loose. We hated it. Asking for time off was always calculated even thought we put so much overtime in so we did exactly what the article describes - half left, half shut down.

So your best developer delivered zero working features in 7 months, was asked to describe what he was doing and quit? And this is the businesses fault for not letting him keep going for another 7 months? Some places micromanagement is horrible, but at the same time as developers we are taught to break problems into smaller and smaller chunks until the problem is trivial. If a developer cannot break a large task into small tasks and then describe them to the business, that is not a good developer.

You should check out Drive by Dan Pink ( http://www.danpink.com/drive/ ). It closely examines the whys of the process of motivation and the giant role in autonomy in creative work ( programming and a whole bunch of other things ).

Nice piece, just one point. I think they are Schrodinger developers if observing changes their behaviour. A Heisenberg developer you would only be able to accurately determine either where they were sitting or how fast they were delivering but not both :)

TOO MUCH DRAMA! Good PO's won't bash "Investigate X Technology" stories. Good developers won't try to build a DSL trying to find a silver bullet to ALL problems. Good teams have a good balance between "turn this code into art" vs "lets release this". Don't need to use Jira but using a whiteboard to track progress is the bare minimum.

Very late reply, but I'm missing something from this post: don't you see at all where you went wrong?

It seems like you already got to the conclusion that a software developer is not an underling who's only waiting for orders. So why did you let others turn you into one? Why did you let a PM make technical decisions for you? Why did you take stories into your sprint that weren't refined enough? Why didn't you break an estimate when you came to the conclusion you needed additional refactoring to cleanly implement it? Why didn't you set a task/story back to TBD when you noticed it didn't make sense?

As a software developer you steer the process just as much as the PM does. Non-developers don't understand how your job works, so you can't let them dictate the conditions of your work. At the same time you need to be aware of the conditions of others' work. As soon as you someone as working against you - be it the customer, the PO, the PM whoever - you already lost. That's what it means to be a professional. And sometimes that means you don't shy away from conflict.

Yes, all the others didn't work towards the goal and the best of their respective companies, but you didn't either.

"Stakeholders" That's the bit that has to work here. The ticket is not the goal, the working product where your new features are being tested by your customers is where it's at. If your developers can't see the vision of the business they can't contribute (other than to adequately code to someone else's specifications - you need those types too!). You either need the business to better translate the value, or better developers.

Ok, I'll get ranted for this, but... developers deciding on usefulness of features? Sorry, but that's a straight way to get a product being an unsellable bag of cool features. I've been in technical sales for over a dozen of years before recently moving into PM and I've seen several really nice products going off the market because of lack of proper use cases, stories etc. I'm sorry, but I dont see devs doing market research, talking to investors, and doing all strategic business planning related to prod mgmt. Not because they're not capable, but simply because they wouldn't like it (and typically, thanks to their PMs, are not aware of). C'mon, sales and marketing? Show me a Dev being fascinated with it ;) And at the end the product must sell to get us all paid.

Business defines problems, devs create solutions. Any of sides getting too much into competencies of the other makes things messy.

One more thing - that was already mentioned, but a good pm would redefine "that feature" to free up his team from being stuck there. Taking care of the team is one of duties coming with "manager" in PM title...

@other anon: "developers deciding on usefulness of features" Not deciding, of course not, but a developer is an advisor and has to take this role seriously. I'm speaking as senior and technical lead here, but when a junior sits in a planning, I expect him to think about every story all the same. It's a developer's job to spot bad features, i.e. features that only solve a problem the customer thought he wanted solved. Ask the painful questions that leave the feature crying in the backlog. That's why user stories are so important. You need to tell developers why you want it to give them a chance to do their job. And if you don't, they need to squeeze it out of you.

We can all discuss at length what role each "participant" in a software project has, but what it IMHO boils down to in the context of this blog post is: be professional, and if others are not professional in your project, be professional nevertheless.

The main objective is to produce a good product, all effort spent on trying to predict and control how long it will take is effort diverted from actually doing it. The cost of better delivery estimates is later delivery.You can and usually do need to put in a bit of effort towards planning and tracking progress, but the nature of the job makes accurate long term predictions hard, and the point of diminishing returns is quite low. Deathmarch projects often have diverted so much effort into managing the process that they can never make progress at all.

I understand and appreciate the feeling of a need for autonomy after an experience as described above. But I think "autonomous" is the wrong label to send to the management team which is confused about leadership style and lost in the search for methods that deliver signs of progress.

I've been following a project from the executive chair, where the manager was not capable of managing the project. The requirements were comprehensive, the PMs knowledge deep, the budget and time sufficient.

My guess is that the project failed to meet its goals because the PM didn't engage the developers sufficiently in the system-engineering part. That meant that the dev-teams output lost track of the bigger picture and spend too much time on writing code that got deprecated rapidly. The dev-teams creativity without appreciation of the PMs bigger picture was what produced problems.

This became evident when I started to investigate what the product-evolution-tree looked like and asked for the minimum viable product. The PM had failed to communicate this to the dev-team, and the dev-team had failed to stop the PM asking for deliverables prior to understanding the product-evolution-tree.

The devs started working on something they didn't understand. That is autonomy without engagement.

The PM accepted their autonomy without having ascertained their engagement.

The problem was interaction: The process did not leverage on the strength of the individuals

The solution that we found became stupidly simple: The PM was not allowed to issue a new task on the task list unless he had a user-test for it. If he couldn't come up with a test, he had to sit with the principal software developer until the test or tests where made explicit.

To limit the autonomy amongst the devs, their tasks were obviously limited by the tests, but also by architecture as every module had to do one thing and do that one thing well.

If they wanted to store data, there was a storage service (a configured db with a client). If they wanted to send messages to other services, they had to use a messaging framework (a resilient queuing system), ..., and if anyone developed anything new it had to have its own generic purpose.This generated the right level of abstract inheritable modules which leveraged on the strengths within the team. Mathematicians came up with clever algorithms. System engineers hardened the interfaces and devOps figured out how to deploy these in a resilient manner. If anyone started on a new "thing", they designed it using the knowledge of all the other team-members, by asking "hey, do we a module that does X?"

This self-engaged collaborative approach resulted in new features every 90-360 minutes, which steadily made it into production as the test suite ran non-stop.

The point:

TL;DR - Autonomy without engagement is just as bad as micro-management.

Why do you care saving somebody's else ship? As mentioned in the comments above - you (me) the developer are just resources and this is in 99% of the time. So why at all bother doing it the right way when you are doing it for somebody who doesn't give an f for you (and this is normal, you don't give an f for them either)? Why take the responsibility? What is your benefit for doing it? Don't get me wrong here. I love software development. I've been programming for more than 20 years (started in 12) read countless books/articles and information of how to grow software and for software craftsmanship. So my point is unless if you are doing it for yourself don't throw too much of your own life into it. I've been trying to persuade tons of PMs and business owners that they just NEED their code refactored and well structured if they want to be able to sustain the product in (the no so far) future. No one listen, and guess what - it is not your money on the line - it's theirs. Project fails you move to the next one (project or employer), you get your paycheck. If you want to do something meaningful - START YOUR OWN company (don't leave your job yet), come up with a product that you LOVE working on (in fact this is no longer called work but fun), let the PM micromanage the project at work, inflate your estimates to the roof, work 2 hours a day for the employer and 10 hours for your own project (your employer is now effectively paying you to build your own project), start selling your product, build some customer base, start selling your product even better, than quit your employer and become an employer yourself (and become a better one).

P.S if you're the only one that can understand the code (because you did your best to make huge mess (you've read: How to write unmaintainable code :D ) you'll be the most valuable dev resource in the company :D and guess what the most valuable resource must be paid tons of money ;) and can't be easily replaced by somebody else. You got the point.

I think you are missing the point. By forcing the developers to push spaghetti code via harsh timelines and never allowing for refactoring, they are adding massive development hours and shrinking their margins overall... while also running the risk of losing customers / clients by the amount of bugs introduced. This becomes amplified if you're constantly cycling developers. A company that understands good development processes stands to gain a lot. If the product isn't cash flow positive, but it's being created according to the requirements set forth, it's a problem with the product, not the development team.

To be fair, I work in an environment where scrum does actually empower the people who do the work. It requires management, execs, product owners, and scrum masters to all step into the shadows, and that's hard for people to do. Letting the developers make decisions about designing the product and control their own workflow is the goal.

Great post. I suspect I'll forward it to my boss the day I resign. It seems to me that there is another huge gap created by this sort of development management.Agile say fix the problem in front of you, but most problems are at heart systemic. The number of times I've fixed a bug with what amounts to a hack because fixing the cause is seen as "beyond the scope of the ticket". I think some version of agile can be useful during bug-fixing periods, or for maybe 2/3 of a large team during the middle of a big build, but (particularly for completely new projects) you need dedicated framework devs who maintain core consistency, refactor to correct issues, and generally just build all the tools the agile devs will rely on.A classic example of this is security. Initially (before release, in the dev environment) security is irrelevant. At some point someone will decide a page needs to be secured, someone will throw together an access security model (totally ignoring data security), and start securing pages.That's not how to do security. Security (if it's relevant) should be the first of the foundations an app is built on. Yes, it means nothing to show at first, yes it takes one of "those" developers to write it, but with it under everything, it ceases to be an issue. Without it, it will always be incomplete.Agile encourages the idea that everything can be broken down into manageable chunks.If a manager is a person that thinks 9 women can have a baby in 1 month, then an agile manager is one who thinks they can allocate a different body part to each woman, and expect a fully assembled baby at the end.

Someone may have said this, it is hard to catch up on all the comments but let me just boil this down to a core argument outline:Intro1. Describe method onea. Method one made friends happyb. Method one made clients panicc. Method one worked for the beginning but ultimately failed badly (see 1b)2. Describe method twoa. Method two made friends sadb. Method two made non-friends happyc. Method two made clients ... unpanicked yet unhappy?d. Method two was not successful (support this)Conclusion (use method one)

I'm unconvinced though I started clearly on your side wanting to agree that autonomy provided more productive work than micro management.

Both methods failed, for different reasons, few of which have been analyzed and fit none of which convincing solutions have been offered.

The issue of who is or is not happy in these methods' environments seems to have no bearing on success. Nor, apparently did the autonomy to whittle away idly on a DSL when the aim had no clarity to the implementors; did the DSL sound like exactly what was requested to the party asking for its needs filled? Considering the response, I'd take it as they absolutely had no idea what you were even trying to do.

I've worked in Scrum environments many times. Mostly I would say they are very hit and miss with regarding to predicting task time. Few people actually track time correctly. They don't remove all the non project time from their calculations. Some use a fudge factor to account for lost time. In my experience my scrums which model the dev time correctly have really given the team a feeling for what they can develop in a given set of time and after a few sprints start to get very accurate. Those that don't frequently slip delivery and project feels wobbly and lost in comparison as a days dev time is frequently a half day or less in some environment.

You have to model time lost to sprint planning, scrum meetings. holidays, training recruitment etc...

Where time is modelled incorrectly I've felt the hole scrum magic to be missing and people come away wondering what it's about and what the benefit is....