I consistently underestimate my work. I'm gaining a bad reputation for it. I want to get past it.

I think the main problem is that I don't fully outline every step that needs to be taken. The first development agency I worked for wasn't big on this. They preferred to just get started, but they still wanted reliable estimates for their clients.

Can you give some more info as to what ways you're underestimating? Are you not including all tasks, not giving enough time for the tasks, not charging enough for them, etc...? In past projects, in what ways did you underestimate? There's a big difference (and different ways to correct) between not identifying all the tasks, and being too optimistic in estimating durations.
– Trevor K. NelsonAug 23 '11 at 21:36

One example is the current project I'm working on. I guessed it would take around 2 weeks and that was 4 weeks ago. I didn't stop to really create a lsit of tasks until a week or so after starting because my boss wanted to get it working since the concept is so simple. BUT, I'm learning a new language and framework().NET MVC 3 and C#).
– BrendenAug 26 '11 at 23:23

1

I think I'm just overly optimistic about things getting done without problems. I also don't see to take enough time to plan thoroughly(I've never had any mentors or peers that devote enough time to planning, they all think it's best to just jump in which I think is completely wrong based on my experiences). I'm reading the Mythical Man-Month now since I discovered it on Amazon and it's helping me a lot.
– BrendenAug 26 '11 at 23:25

PS - I'm a developer not a PM, but I have to estimate the work I am assigned.
– BrendenAug 26 '11 at 23:35

For an individual solution to your problem, a typical technique is to "multiply them by π". For a more systematic group-based approach, I would suggest techniques such as Joel Spolsky's "development speed" or the Agile story points as said in Erin Beierwaltes' answer.
– Daniel DaranasApr 15 '13 at 7:15

12 Answers
12

Humans are no good at estimation. This is why Agility frameworks like Scrum use things like Story points, velocity and/or throughput to help the business and teams better PREDICT how long its takes to develop features.

I wouldn't beat yourself up about your estimates. I spent a lot of years doing traditional earned value and knew in general how much people were off, but it was never used appropriately by the business and we always ended up playing schedule games where we would try to tweak this and that and waste a lot of valuable development time.

I would suggest doing some research on Agile, Scrum and Kanban and having a conversation with the person asking for these estimates about the difficulty and futility of your search for improvement.

I'm going to select this one as the best solution. You nailed it on the head: Humans are no good at estimating(esp. with such complex matters). I try to use a practical version of Agile/Scrum and ithink it is the best way to approach project management. Thanks.
– BrendenAug 31 '11 at 19:51

5

Humans are great at estimating! Predicting the future with absolute certainty, we are not. And these are two vastly different things that too many of us in projects confuse! Estimating is a shot gun blast, not a single bullet from a sniper's rifle. And, in fact, since our world is quite non deterministic, a shot gun blast as far more representative of our reality than a single shot. Unbelievably myopic to suggest humans are bad at estimation.
– David EspinaSep 2 '11 at 17:46

If you have a bad reputation for underestimating, do you also overestimate as often? If so, you may not realise it - we're wired to notice the things we do wrong, more than we celebrate the things we do right.

If you're doing this, it may be that you're simply picking work with a high degree of uncertainty. You may even be underestimating because of this even if you don't feel that you ever overestimate to match, since work expands to fill the time available, and there will be plenty of people to take advantage of any free resources you have.

You say you want reliable estimates for your work. Why do you want them? To gain the trust of your clients? I know plenty of agencies and individuals who underestimate deliberately in order to win contracts, then make it difficult for the client to change their minds about anything - and people will always change their minds.

Are you more flexible than others? Are you more responsive to change? Do you keep your projects responsive and able to change? Do you share the things that you learn as you do the job? Do you keep options open until a decision is really needed? Can you sell yourself on those skills, instead of your ability to estimate?

Are you trustworthy? Are you genuinely trying to give a good estimate, based on what people have said without contingency for learning and discovery, when other people are padding theirs? Do you refuse to cut quality when the deadlines are tight?

Do you need to provide the estimates yourself? Could you work on a project that someone else has estimated? What skills would you need from someone else to be successful? Would you be willing to share that success with them?

Plenty of industry evidence shows that individuals who concentrate on their strengths are more successful than those who concentrate on their weaknesses. Where are your strengths?

I understand history based estimates but my problem with that is that one feature in a project doesn't relate to another. I guess it just gives you a better idea of what it'll take. Thanks.
– BrendenAug 23 '11 at 19:39

+1 for keeping track of historical data. I would strongly stress using them in learning how much you underestimate. Usually we're pretty consistent with the scale of our underestimation.
– Pawel BrodzinskiAug 24 '11 at 10:26

Brenden, if you are having trouble comparing apples to apples, make the work packages less specific e.g. instead of "implement script x" make it "implementing scripts (being a bucket of scripts)." Also, recommend that the agency use a project manager. Developers should focus on being good developers. Project managers should focus on estimates.
– Mark Phillips♦Aug 24 '11 at 16:25

A developer's estimates are a single point of data that should be used along with other schedule/effort estimate data to produce the final schedule. Info like your past history of underestimating and comparing it to actual would factor into the estimates a pm gives to the client.
– Mark Phillips♦Aug 29 '11 at 13:09

Are you talking about your estimates or your target? These are two different things. We set our targets using a single-point value, that is we say we will finish this task in x days with y dollars. That is a deterministic target. Sadly, we live in a non deterministic world, where our performance for said task will be in the neighborhood of x-a days to x+b days with some probabilistic curve that sits on top of that. And your performance under this curve is largely dictated on random events--stochastic noise--over which you have little to no control.

So to really evaluate your performance on a single project, or even 10 projects, as to whether you did any good or not cannot be done comparing your actuals to your target.

Here is why:

If you estimate a task will take between 10 to 25 days with a most likely of 15, and you consistently set your target at 12, you are creating an aggressive schedule where most of your likelihood is beyond 12 days. What is better? Setting a target at 12 and getting 15, a 3 day unfavorable variance, or setting your target at 23 and getting 23, a zero day variance.

The unfortunate fact is PMs are evaluated on their performance against targets, no matter where they fall on the probabilistic curve. This incents PMs to set their targets on the fat and happy side of the curve, thus increasing their likelihood of success. This gives the PM a win, but customers lose because they end up paying more.

So, while breaking down your work and using various techniques of estimating are great, I would suggest that all of your estimates, including the contribution of your staff, be in ranges versus a single point, if you are not already doing this. This will give you the best insight on your schedule/cost risks and, if you bust them, will give you confidence in your true performance overall.

How about creating your own estimates, then getting a peer review - maybe having a discussion with someone whose judgement you trust? That way you will learn from the experience of others, and while you don't have to accept their input (they may also be poor estimators), there is a high likelihood that you will have a discussion and that will lead to a better overall estimate at the end. And if you can get a couple of different people to help you with your estimating, then you have an even better chance of including everything that you need to include, and making the estimates realistic.

The whole idea behind building a WBS is to create bite sized pieces of work that can be more accurately estimated. If you're having trouble estimating, then break down the tasks you have listed even further until you can get a better handle on what needs to be done.

Estimating is hard - don't beat yourself up about it too much. Work on gathering data that can help you build more accurate future estimates.

Well... maybe not fully outline each step but at least break the work down into discreet chunks that are easier to estimate. For example instead of Login Screen - 1 day... try breaking that into UI (just the pretty part), back end code to handle the UI, code to communicate with a security service, etc...

It can be a bit of an art, because you have to figure out the degree of decomposition you need to get too.

Another approach (based on the Personal Software Process) is to try to track metrics on your self. Get some hard numbers as to how long it takes you to do various parts of a task. Then use that for base estimation in the future.

Seems you and Mark think alike. So how do you relate your metrics from one project to another?
– BrendenAug 23 '11 at 19:39

Varies of course but I tend to group things into UI, security, database, etc. Essentially categories common across the development life cycle (including deployment/installation, testing, pilot). so as you gather data you can see where the similarities are.
– edgaralgernonAug 23 '11 at 19:45

From what I've seen in my field the rule that 80% of requirements take 20% of the time to implement holds true. Where we get our problems with preparation timelines is that the amount of time to go from 80% of what the client wants to 90% to 95% to 100% increases exponentially. This can totally blow our time estimates, as the search for perfection takes a loooong time before it gets called off out of frustration. You can try to avoid this by:

Planning in detail to your planning horizon only and communicating ASAP when the overall project plan is no longer realistic. If this happens make sure you have a good reason that the client can understand the issues even if he isn't happy about it.

Keeping customer expectations aligned with what you are going to deliver at each stage of the project.

Being open about your contingency reserves so that you are providing your client so that they can have an early and late date for delivery. Also be honest with yourself to avoid treating the contingencies as extra time - always strive for the early date.

Learn from your past experience - which I think you are already trying to do based on the fact that you are asking the question in the first place :-)

Going over your estimates or under-estimating? How would you handle the situation if the problem were short estimates, and not long execution?

Remember Tom DeMarco's discussion on estimates? A good one is as likely to go under as over, whereas the earliest time with a non-zero chance of completion is not an estimate at all. I don't pretend to know what your estimates are like or how you arrive at them, just wanting to offer an alternative definition of the problem.

I think I'm starting to understand there is no science to hard and accurate estimates. I guess the real question is how do you better guess at the effort something will take?
– BrendenAug 26 '11 at 23:27

We tend to be better at comparing relative sizes than guessing in hours. Take a task you've done, and call it "one" or even "two". When you see a new task, compare them. Does it seem twice as big? Half-again as large? Give it a new value. Often we can get a better order-of-magnitude estimate that way. I like using the fibonacci series -- two is twice 1, three is half-again 2, etc. Don't think in hours/days/weeks until you have a sizing. When sizing feels right, calc estimate v. actual hours of original. Will still be wrong, but maybe less so.
– Tim OttingerAug 30 '11 at 2:35

Better yet, what if you did good work quickly, and then didn't get measured against your estimates at all? Estimating is 80% waste.
– Tim OttingerAug 30 '11 at 2:39

In the interim, if people are getting pissed with you, double up on everything. If you think an hour say two, if you think two weeks, say four.

Its better to under promise and over deliver. You are always going to have to battle/bull shit on a job, so do it at the start rather than the end. Then, when you get the work in early everyone will be happy.

I run an agency and it is about reliable estimates for customers but it's more about keeping them happy, that's the big goal. If I tell them two weeks, and deliver it in 1.5 I can either discount the work or add to the work make it better, everyone wins.

Development is a service based industry, not product based. You need to remember that there is a difference between duration, estimated work and actual work.

As a developer who is now a manager, estimation is hard. As Scott said, you really mustn't beat yourself up about it; it's just hard. Frustratingly, the way to get better at it is through practice.

Personally (and your brain may well work differently), I find that the bigger something is the less accurate the estimate. Break everything down so that you aim for half-day chunks or smaller -- instead of "build login functionality", try "build login form + build lost password form + build login database" and so on. For some context, I estimated a Facebook application for a client today and the largest line-item in the spreadsheet we use was two days, when the project duration is two or three weeks build.

Remember that nothing ever goes smoothly, so take a guess as to how risky each task is and vary the quote accordingly. For the sake of argument, let's say that low-risk tasks should be inflated by 10%, medium risk tasks by 50% and high risk tasks by 150%. (I'm making these numbers up as I type, just pick something that feels right, then increase it some, to make sure you're not being too optimistic again.)

We use our bug tracking software (Jira, but any would probably do) to track work items too, and to log time against them. (This has the advantage of allowing project managers to see how you're doing against their expectations.) You don't necessarily need to do all of that, but it's a very good idea to keep track of your estimates, so you can go back and compare your performance against them later. That will allow you to gain an idea of the kinds of tasks that you underestimate and by how much.

The main thing you need to work out, I would guess, is in what way are you underestimating? Are you just generally being optimistic, are you underestimating the complexity of certain tasks or are you not taking account of the things that might get in the way (other work coming up, that meeting you need to allow a couple of hours for, that hangover you're gonna have after taking your partner out for Valentine's night, the time you're gonna have to spend googling how to get MVC to play ball with the data model you're passing in from this controller, whatever). Finally, what about amends time? In my day job, all our dev work goes through a QA team, who come back with bug reports. Those bugfixes are usually around another 15% of the build time, all told. So a project that's a week's build will probably have another day of bugfixing once it comes back from QA.

All that said, for the project you're on at the moment, stop where you are and review what's left to achieve. Divide that up into discrete tasks and re-estimate it. Then take a look at those risk factors and inflate your estimates accordingly. Then, when you're done summing that up, add some contingency (we apply 30% contingency to most projects, because shit just has a habit of coming up when you don't want it to).

You mention that you have some colleagues, but that they're not big with the up-front planning. Do you think they would be able to help you validate those estimates? Remember, when asking them, that they will work at a different velocity to you, so if you say that something's a 2-hour job and they think they could do it in an hour, what's important is how long it will take you to complete the task.

As I said at the start, though, don't beat yourself up over it. Development estimation is hard. And building something in a new context (a new framework or language or whatever) is doubly hard. Good luck with it. And don't worry: C#, .Net and MVC do get easier. And so does estimation.