Pages

Sunday, January 15, 2012

Estimating tasks in hours is local optimization. Stop doing it!

This weekend I was listening to Eliyahu M. Goldratt's "Beyond the Goal" where he expands on the Theory of Constraints. In chapter three I listened to him explain how estimating in hours is in fact local optimization and therefore "idiotic". Here is a portion of that chapter loosely transcribed:

"The way to ensure that the project will finish on time is to ensure that every task will complete on time. This is the local optima policy that leads to a huge conflict. Placing a person on such a project jeopardizes the most important thing for them - their own image.

Suppose you are this person and you are asked how much time will this task take? You are extremely reluctant to give any number. And when you give that number the Project Manager will try to squeeze it. Why are you reluctant? Because you know that the number you give is an estimate but that it will be converted to a commitment - because the project needs to finish on time.

So what situation does this place you in? This common procedure of turning estimates into commitments puts you into a huge dilemna. Why? The objective of every person is to be regarded as a reliable person by your boss, by your peers, by people. But what is the meaning of being reliable? For example, if you give a commitment and you don't meet it - once is ok. But if you give a commitment and many times don't meet them do you expect to be regarded as reliable? Of course not. So one of the conditions of being a reliable person is to meet your commitments.

Also - if you are fighting for something (an estimate) and then it turns out that you have exaggerated by a mile? If it happens once, fine. If it happens frequently, then you will get the name of someone who exaggerates wildly - so kiss goodbye the image as someone who is reliable. In other words, another consideration for being a reliable person is to not be considered as someone who exaggerates. Now look what is happening when I come and ask you how long it will take to finish this task?

Now look - whatever you do, you are doomed. This is a huge personal conflict. How do we in reality deal with this conflict? Let's face it - this conflict is so important we must find a way to break it. We want to not only be reliable, but to be considered reliable. How do we do it? We cheat.

So we finish our estimates on the nose and sometimes a little more. They become self fulfilling prophecies. If there is a danger of finishing too quickly we add some more tests, do a more thorough job, we add another function in, we check things a little more thoroughly, and we finish on the nose. Do you see what's happening?

Due to the result that our local optimization turns any estimate into a commitment, we have developed the habit of giving estimates which includes in it the safety. If murphy (i.e. Murphy's Law) doesn't hit, we waste the time rather than giving it to the next link in the chain or project. If we do give it to the next link then we are declared as exaggerating and next time they won't give us the extra time. So look what a horrible situation this is. And this... is what kills the project. This is totally idiotic"

When you read this story the first time you will likely notice there are several examples in there of how not to run a project, of poor leadership, and his concluding statement is a little harsh. As many commenters have pointed out, this style of project management is not similar to how we typically run agile projects. However, at the core of the story is the desire of each person to be reliable by meeting hourly estimates whether they are created by the team, by yourself, or by someone else. When I first listened to this argument I was fascinated by that part of it. So, if that part of his argument is correct, then what alternatives do we have if we need to estimate? (This paragraph has been updated based on the comments - thank you)

If your project requires an estimate:

Stop estimating tasks or user stories in hours - hopefully the
argument above has already given you enough reason to at least consider
this 'radical' change.

Don't estimate user stories in hours or convert relative points to hours. OK - I do understand that it can be valuable to look historically at hours vs. points, but don't use this conversion for guessing how long a task or user story will take.

There is some evidence
that due to variability, counting the number of stories completed in an
iteration is of equal value to counting the number of points completed
in an iteration (velocity). This would help keep the focus off of hours
per user story or task.

Where possible, avoid asking how many hours are left for a task or user story. I understand why and how this information can be valuable, but understand the risk you take when you ask this question.

If possible, estimate at higher levels (epics) rather than lower levels (user story).

Give broad project estimates with ranges like 1-2 months, 3-6 months, 1-2 years instead of doing detailed estimates. Most projects don't need a detailed estimate in order to determine if they should be completed or not. For many projects on our list we already have a good enough guess about their size to understand their value compared to the cost. (Unfortunately, as a consultant that usually doesn't include the projects I work on).

This logic also suggests that relative estimating might be dangerous. I still have to think about that one as there are significant benefits to exercises like planning poker even if estimates are not created as output.

If your project does not require an estimate:

Do a little song and dance!

In either case, continue to break down your project into smaller projects and small deliverable pieces of value and working code. Build the high priority pieces first in an iterative fashion. Make a decision about whether the project is done after each completed story or each iteration to avoid the problem described above - the temptation to finish on your estimated completion date. Continue measuring your progress using completed stories but consider using cycle time (average time to completion for all stories) vs. velocity (average number of points completed in a given time frame) to avoid some of the local optimization that could occur for each user story.

For further reading on the Theory of Constraints, check out any of Goldratt's books. I highly recommend starting with The Goal which is also available on iTunes Canada as an audiobook for $2.95.

17 comments:

I highly recommend Goldratt's books. So far I've read only The Goal and Beyond the Goal (actually, not traditional reading, per se, but I've listened to the audio versions of both and am currently in the last third of Beyond the Goal) and I look forward to reading some of his others.

I have been thinking about the ideas posted above for quite some time, and as I am currently taking a course in Project Management I have a whole new perspective on them.

The part of Goldratt's argument quoted above that most struck me was the conflict that PMs place developers in when asking them for estimates. As I read it, and as I have lived it now from both sides, the combination of PMs trying to over-specify a project schedule intersects with the goal of developers to be perceived as trustworthy, knowledgeable, and reliable, and also intersects with statistical variation, cover your ass (from both roles), and dependent events to encourage inefficiency and "slippage".

And yet both PMs and developers often find themselves in the position of having to answer some form of the question "how long will this take?"/"how much will this cost?"/"when will this be done?"

So how do we PMs avoid placing developers in this conflict? And how do developers placed in this conflict find their way out?

My current approach is to carefully examine each project management tool and technique I come across to assess what it is trying to achieve and evaluate it in terms of it's assumptions and the behavior that it is likely to encourage.

Your response has two parts. In part 1 you summarize your version of his argument and then eloquently declare it bull cookies. In part 2 you start responding to his legitimate concerns with "what I try to implement on my teams that address some of Goldratt’s concerns". Perfect.

Let's just focus on Part 2. Shared Understanding, Technical Leadership, Open and Honest Communication, plus Creating a Culture of Delivery. Yes! If you read his work you are making his overall point very nicely - "Delivery is a good thing, it should be encouraged, and it should be celebrated!" A focus on delivery is a focus on throughput (global optimization). A focus on hours per task is a focus on local efficiency (local optimization). This is true whether you work in a Dilbert cartoon or not. So you both agree on this at least.

So is estimating based on hours bad? No. Estimating tasks and stories (local optimization) based on hours is bad. Instead of a bell curve (some estimates will be under and some over) you get a long tail (some estimates are over and most of the rest are exact). Nothing you have said changes my opinion of that. Estimating in points is a much better approach because it moves the focus to delivering value and increasing throughput rather than meeting your hourly estimates.

I is bull cookies! PMs are not all evil time crunchers, developers are not frail or timid. The text you quoted makes assumptions of what the PM is incentived by (hours and estimates), and that the developers are working isolated from a team and individual estimates are requested on a dev-by-dev basis. While I don't disagree that this happens, its not a foundation to dissuade the original question of whether storypoints are better than hours.

In that environment, estimating in hours, story points, or sheep doesn't matter because the deficiencies of the environment are the root cause of project failure, not the estimating unit of measure.

If I say a screen will take me 8 hours and it takes me 12, then a screen of similar complexity will also take me 12 hours.

If I say a screen will take me 3 story points and I've associated that with my initial 8 hour estimate, and it takes me 12 and I then call that 5 story points, what have I done? I've just created an abstraction over actual time deliverables. Here's how the convo with the PM goes:

Me: That new screen will be 5 story pointsPM: So 12 hours?Me: Yes.

So why not just say 12 hours?

Again, its all about culture and environment. Make the focus about delivery, make the team aware that accurate estimates are important, and then focus on getting stuff done.

We agree that this doesn't have anything to do with evil time crunchers or frail and timid developers. We agree that culture and environment are important. We agree that a focus on delivery and getting stuff done is important.

We don't agree that estimating in hours is dangerous. So why not just say 12 hours? The argument for that is above and you disagree with it. I'm ok with that.

I haven't had the chance to read the Goldratt book and I can't say I fully grasp the "local optimization" concept. My point on our twitter discussion was strictly pertaining to translating points to hours. I find that in many cases, teams don't fully understand the purpose of measuring effort in relative complexity (story points, shirt sizes, whatever). We're all trained to think in hours and it's very difficult to get your brain to stop translating everything into hours.

In my experience, measuring in relative complexity only makes sense when used with velocity. Measuring in hours (or translating points to hours) is subjective and always relative to the estimator's frame of reference. Group estimates can mitigate this to a degree but it's still a subjective measure. Velocity, on the other hand, is completely objective and takes into account ALL dynamic elements (variance in team members output, skill levels, team turn over, technical complexity, etc.). One could say: "the velocity doesn't lie". From a change/risk management perspective, I am fully committed to this approach and in my experience it's a more effective means of ensuring overall project success than thinking strictly in hours. However, from a budgeting perspective, I still have trouble understanding how you can avoid translating to hours when asked to estimate the cost of a project. So, to say that estimating in hours is bad or idiotic on the whole? I'm intrigued by this assertion.

Goldratt seems to be very respected in this field and from what I've read, his "Theory of Constraints" is widely accepted and taught in hundreds of schools and companies so there must be something to it. I just can't see how you can avoid estimating in hours in all scenarios. For example, a product company may publicize a release date as part of a pre-marketing campaign and, in a competitive market, missing that release date can be catastrophic. How can you avoid keeping track of progress in hours? If you're falling behind, yes, you're probably going to have to take shortcuts and fix stuff later. Ex. Apple launched iPhone 4 with a defective antenna bcs they knew that they would lose a lot more by missing ship dates than than pushing out an imperfect product. The defect was well publicized, yet they still broke sales records. Later, they gave away free bumper cases and everyone was happy. In their industry, timing is everything.

Regardless, this is quite a fascinating topic and I look forward to learning more. One thing is for sure, it's definitely not "Bull Cookies" (although this did make for a very sexy blog post).

He isn't saying that estimating in hours is bad or idiotic - only that estimating in hours for tasks (or user stories) is bad and idiotic.

Translating velocity into hours for budgeting and planning the future makes a lot of sense. For your example - when trying to identify a future release date you can translate your current velocity plus the remaining points into a best-guess date without having to use hours. The hard part comes when you don't have a velocity yet - but estimating hours per task is the same amount of guess work as estimating the number of points per week.

Thanks for the response Terry. Both you and D have focused heavily on the bad management in the story above. I agree 100% that the story includes bad management and unfortunately that takes away from the central theme - that is my mistake. Now take out those comments you and I don't like from the story (estimates to commitments, ensuring every task will complete on time, idiotic, estimating alone, task vs. stories, etc) and read it again.

What is your response to the rest of it - the part that describes how estimating in hours causes people to locally optimize in order to be seen as reliable?

Some thoughts:

1) When you expose the hours to someone on your team for a story or task, they are caught in the trap above even if they haven't created the numbers themselves.

2) When you expose the hours to someone on your team, you have just negated one of the benefits of planning poker - that a task you might complete in 8 hours might take me twice as long due to skill and experience.

3) When you expose the hours to someone on your team, are they looking at those hours as development hours only? test hours? deployment hours? analysis hours? all of the above? If they are only doing the development, how many hours of that total should it take? In this case you may again be negating the benefits of story points by breaking them down into smaller chunks - local optimization. Granted - you may be tracking time for each story per role, but even then there will not be an even distribution per role per story point. Stories of the same size do not have the same amount of analysis/design/dev/test/deploy effort.

4) I too have witnessed individuals being thankful to know the hours associated to a task/story. In fact I have been one of those individuals. But look at this in context of the story above - they have now been given their 'reliability' number and will strive to meet it - Parkinson's law in effect (Work expands so as to fill the time available for its completion). Add in Murphy's Law and most tasks/stories will take 100% and others will be more. Also, in the story told by "The Goal", many people are happy to continue with (and fight for) locally optimized cost accounting in the fictional factory but their fervent belief in cost accounting doesn't make it effective. (If you haven't read the book, then this reference might not make sense).

In conclusion - yes we all agree that there are elements of bad management in the quotes I shared above from the book. I have learned my lesson and I will take them out next time. The central theme of the story above is about a person's need to be reliable (and not about frail or timid developers) and how estimating in hours affects the end result. In that case the story nicely illustrates why estimating in hours is dangerous.

Sorry Steve, I just had to respond because I think many of the items again point again to bad management.

1) Bad Leadership. If there aren't involved in the estimates, they will be allowed to re-estimate and provide new estimates.2) You don't expose hours until after planning poker is completed.3) Bad Leadership. If it isn't clear what you are estimating that isn't a problem with the unit of measurement. That is just awful communication.4) Pet Peeve. Murphy's Law and Parkinson's Law are not actually laws. They are theorems that have gained popular appeal. Nobody has ever proved that work expands to fill the allowed time in all situations. I could again point to bad leadership and not following YAGNI if this happens...

I agree that estimating in hours has challenges and dangers. But I think the benefits can be worthwhile if the team understands the pros and cons...

I didn't quite understand or agree with all of Terry's replies in #1-3 above so we took the conversation offline to make sure we understood each other. The result?

Terry likes to use hours to help set expectations but is careful to have people remember that different people have different velocities and stories have variation in themselves. Terry understands Goldratt’s reliability dilemma but asserts that a good team and good leadership can overcome that – especially since there is value in knowing the hours in some circumstances.

Having worked with Terry and knowing what kind of leader he is, I can live with that even if I wouldn't necessarily use the same approach.