Slashdot videos: Now with more Slashdot!

View

Discuss

Share

We've improved Slashdot's video section; now you can view our video interviews, product close-ups and site visits with all the usual Slashdot options to comment, share, etc. No more walled garden! It's a work in progress -- we hope you'll check it out (Learn more about the recent updates).

snydeq writes "Fatal Exception's Neil McAllister discusses why code analysis and similar metrics provide little insight into what really makes an effective software development team, in the wake of a new scorecard system employed at IBM. 'Code metrics are fine if all you care about is raw code production. But what happens to all that code once it's written? Do you just ship it and move on? Hardly — in fact, many developers spend far more of their time maintaining code than adding to it. Do your metrics take into account time spent refactoring or documenting existing code? Is it even possible to devise metrics for these activities?' McAllister writes. 'Are developers who take time to train and mentor other teams about the latest code changes considered less productive than ones who stay heads-down at their desks and never reach out to their peers? How about teams that take time at the beginning of a project to coordinate with other teams for code reuse, versus those who charge ahead blindly? Can any automated tool measure these kinds of best practices?'"

Refactoring and metrics are often rarely agree. I remember one job I came into, I was given this steaming horrible piece of php (ugh) mess to fix up and add a bunch of features for the client. I took one look at it, and said "I need a good two months to fix this code" and demonstrated a series of horrible vunerabilities and fuckups in the code. Realising the vunerabilities (hint, demonstrating security holes in existing code is a REALLY good way to convince management you need to spend some quality time refatoring it) where just plain dangerous, I got given the go ahead.

A couple of weeks in, the manager called me into the office asking "Shayne, this is very strange. You know we take a metric of how many lines of code per hour are written into account here?". "Yes" I replied, "You also know I think its idiotic". The guy said "Right , well your registering a highly negative score on this. I dont even know if this is working right?".

The reality was this code was written by someone who really didn't "get" databases. A table would be loaded in, then iiterated over, and each pass would issue new sql calls for data, which would then be loaded in and iiterated again. I'd just delete all of the twenty+ pages of code, write a half-page winner of an SQL query, drop it into a mysql stored procedure, and select from it, then spending half a page formatting it and displaying it. It was robust, safe, and fast as hell. I went through the whole code like this turning hundreds of pages of code into tens of pages, all neatly commented, scrubbed of nasties, isolating out the design from the logic, putting models into model classes, views into templates and controllers to glue it all together.

And the end result was every day the code would lose twenty-thirty pages of code and develop 2 or 3 pages of terse, clean, maintainable code.

And as an end result, my metrics where fucked. I tried to argue that they needed to look at more sane metrics, like what was coming out of the CI software, and if they wanted to be really clever, things like complexity metrics and the like.

Ultimately however,they still wanted the "codes of line per day" metrics because as non technical people it made them feel like they understood what was going on, but ultimately they didn't. My suggestion to them however was "When developing fresh new code, that number should increase, when refactoring code, it should decrease. But only if the coder knows what he's frigging doing."

Writing for reuse can be excessive, but there are a number of reasons to move in that direction even if you don't intend to reuse that specific code block:

1) Unit Tests. If you abstract your functionality in a way that allows reuse, it also abstracts in for extremely easy unit testing. And unit testing will save you an incredible amount of effort in code maintenance.

2) Consistency. If you follow the same design pattern for all of your abstractions, all of your developers should be familiar with it. This makes it significantly easier for different developers to step into projects as the hopefully don't have to learn another person's style for abstraction.

3) Replacement and isolation. Need to implement a functional change? If your code is abstracted, like it would be for reuse, the functional change is limited to a single block, which is easily identifiable and if you're doing it right, unit-testable.

4) Just in case. Most of the time abstracted code doesn't get reused, and event when it does get reused it's usually a copy and paste job instead of a reference. Even so, if anyone ever does need the same functionality, it allows them to quickly rip off the exact piece they are looking for as opposed to trying to strip out your programs logic to get the tiny bit they want.

Or you could just communicate with your developers, be aware of the work they're doing and judge their performance based on their effective productivity from your perspective. I've heard this called "management" before, but I know that word has been twisted to mean something more sinister as of late.

Or you could just communicate with your developers, be aware of the work they're doing and judge their performance based on their effective productivity from your perspective. I've heard this called "management" before, but I know that word has been twisted to mean something more sinister as of late.

That won't work, because we laid off all the middle managers years ago. Developers are all exactly the same - we just need to know which ones to slot into the critical path.

Or you could just communicate with your developers, be aware of the work they're doing and judge their performance based on their effective productivity from your perspective. I've heard this called "management" before, but I know that word has been twisted to mean something more sinister as of late.

That won't work, because we laid off all the middle managers years ago. Developers are all exactly the same - we just need to know which ones to slot into the critical path.

Sounds like you did it right. Most places I've seen laid off the developers, out sourced that work, and added a layer of management to manage the over-seas team.

Yes, they are objective and quantitative. Managers love that, because being objective excuses them from forming the kind of bias people used to call "leadership", and being quantitative they fit into nice formulas where they can convey no information at all to higher managers that are too insecure to ask what it means.

Just don't expect those quantitative and objective metrics to be correlated with the overall profit of your business.

It seems you never had to deal with a crappy or unfair boss. Brotip: when you find one, document the crap out of everything. It'll be hard for him not to look like an asshole when you make your point with real data and he ignores it.

The fallacy is that any metric (lines of code, number bugs reports closed, mean time to complete fixes, etc.) or a combination thereof can differentiate between the programmers that should be promoted and those that should be fired. The assumption is that "good programmers" write more lines of code, fix more bugs, make the code run faster, or even write code that optimizes some arbitrary complexity metric. In fact, a "good programmer" does none of these things.

Yeah, I've come to realize that nobody understands what a manager is supposed to do anymore. Most people's concept is that managers exist to play the part of the PHB, but they don't do anything useful.

More and more, I think no one even understands the value of having a knowledgeable person make difficult decisions based on well thought-out judgments. People want procedures that they can just give to anyone and assume that the outcome will be the same, and then they want metrics that they can use to measure the outcome without knowing anything about the situation.

The proper role of a manager-- aside from particular responsibilities he/she might have on top of this-- is to understand the people working for him and understand his unit's role in the greater context of the company, and then to eliminate obstacles that prevent the people working for him from performing their roles. This might mean protecting the people that work for you from upper management, it might mean recognizing and rewarding good performance, or it might mean all sorts of other things. But to do it right, it takes a lot of work and it takes good judgment.

"More and more, I think no one even understands the value of having a knowledgeable person make difficult decisions based on well thought-out judgments. People want procedures that they can just give to anyone and assume that the outcome will be the same..."

Don't you see several complaints around here that managers treat developers as gears in a box or stuff like that? Do you really think that is restricted to tecnicians?

Yeah, that's my point. As a manager, it's your job to learn about the people who work for you, to help them understand what they should be working on, to help motivate them, to judge their strengths and weaknesses, and to coordinate/arrange things so that their weaknesses don't keep them from doing their job well. Then you have to weed out people who aren't going to do a good job, and meanwhile reward, foster, and develop the employees who have potential and do good work.

That's a lot of work, but that's the work that a manager is supposed to be doing. Too often they shirk that work and instead treat their employees like "gears in a box". Sometimes it's laziness, but a big part of the problem is that our society/culture doesn't recognize the work/judgement of a good manager as valuable. We instead expect people to be interchangeable, which is a problem.

People want procedures that they can just give to anyone and assume that the outcome will be the same, and then they want metrics that they can use to measure the outcome without knowing anything about the situation.

That's what modern (from 1911) "scientific" management is all about. That's "best practices". That's ISO 9000, TQM, and all the rest of the alphabet soup. You break the work down into a number of tasks. You measure alternatives and find out the best way to do that task. You document that be

Yes we've tried entirely subjective management, that's why we're trying to find objective criteria. Nothing like a manager that can set pretty much any performance rating he likes, that would surely not be gamed or abused. Combine that with most everybody thinking they're above average software developers and you got the scene set for feeling unjustly passed up. Not to mention the total lack of transparency, ability to compare with other teams or to measure the performance of the managers. Neither side is e

That's the end product. This is about the actual writing of code. You can apply any or all of those in respect to the end product independently. A coder could write very neat code, fast code, or compact code, which has any kind of interface or performance.

Modifiability - This captures the extent to which a programmer correctly anticipates future changes that the code will need. It wouldn't be wise to judge a developer on this track record for just a single program, but with larger sample sizes, it could be useful.

Time-to-develop - If you ignore this metric, you're unable to distinguish between a program that gets written in a day vs. the very same program getting written in a year.

But not for correctness? Security? Robustness? Scalability? Flexibility? I think a test class I took suggested something like 15-20 separate criteria. There's a reason you can write whole books on what "good code" really is. And none of those really cover if your architecture is borked, even if the code in isolation is pretty well done.

Whatever you measure will be gamed. Measure bugs fixed, and you will find people wasting time listing each tiny variation of a bug. Measure lines of code, you will get spaghetti code.

It almost seems better to measure a bunch of things and use a secret formula to determine productivity.

A secret formula like, oh I dunno, customer satisfaction (maybe "how many bugs make it to the customer")? Every piece of software is written for a customer (of some sort) so maybe time could be spent actually thinking about how effective the code is AFTER it's used. Speaking as someone who doesn't write much code, but lives and breathes the consequences of good and bad code every day, I can say with confidence that quality would go up if the only coding "practice" was severe punishment of anyone who intro

So, you want to reward Wally? He probably doesn't introduce much bugs...

If his projects are rated as satisfactory by management (this "metric" ALWAYS comes first, no matter what the development style is) then sure. Sometimes the zen of knowing what not to do is more effective than having the energy to code 10,000 lines of something that is buggy and ultimately unneeded.

Coming up with a developer value metric is NP hard. The best you can do is use human intuition (AKA, a formula so secret even the user isn't sure what it is) to approximate a reasonable result and then look at metrics to see why the intuitive approximation is/is not valid. It cannot be proceduralized without hand waving.

Note to managers?: You don't WANT this proceduralized or YOUR manager will replace you with a spreadsheet. Watching your former employer tail spin into the water might provide some brief sat

Whatever you measure will be gamed. Measure bugs fixed, and you will find people wasting time listing each tiny variation of a bug. Measure lines of code, you will get spaghetti code.

Two things:
1. Your point - which is "Tell me how you measure me and I'll tell you how I'll behave" - with the addition that it's so natural/visceral a reaction that one doesn't even need to intend to game the measurement, the behavior of the measured will alter
2. I'm firmly on the opinion that code is value-spent and is not value-created. From this point of view, seems very strange to me for one to use metrics of cost to measure productivity. Efficiency should be measured in how small the number of lines

Lines of code doesn't make sense because it has nothing to do with productivity. Some problems are harder to solve than others, some require more lines of code and are harder that way, some require complex logic. Some problems can leverage existing code, some can not. Some features just require a lot of skillful placement within the existing code. On top of that, I could write something with very few lines of code using a very resource-expensive technique, is that better?

Whatever you measure will be gamed. Measure bugs fixed, and you will find people wasting time listing each tiny variation of a bug. Measure lines of code, you will get spaghetti code.
It almost seems better to measure a bunch of things and use a secret formula to determine productivity.

The important thing to do before trying to figure out how to measure something is to decide what it is that you want.

But what happens to all that code once it's written? Do you just ship it and move on? Do your metrics take into account time spent refactoring or documenting existing code? Is it even possible to devise metrics for these activities? Are developers who take time to train and mentor other teams about the latest code changes considered less productive than ones who stay heads-down at their desks and never reach out to their peers? How about teams that take time at the beginning of a project to coordinate with other teams for code reuse, versus those who charge ahead blindly? Can any automated tool measure these kinds of best practices?

Do your metrics take into account time spent refactoring or documenting existing code?no

Is it even possible to devise metrics for these activities?maybe

Are developers who take time to train and mentor other teams about the latest code changes considered less productive than ones who stay heads-down at their desks and never reach out to their peers?probably

A lot of problems rating developer productivity. First, if a system is that good, then managers won't be able to game it to play favorites. Second, writing code for future use is always harder than writing code specific to the problem. Third, almost any metric is going to penalize a simpler solution. (Keep in mind that once you see a simple solution it seems obvious and everyone thinks they'd think of it. Fourth, evaluating developers well would require making the best coders managers, and that rarely happens for several reasons.

First, if a system is that good, then managers won't be able to game it to play favorites.

This is the most likely reason I see why no performance metric that works would actually be picked up. The ideal rating system for a corrupt manager is one where everyone is rated poorly, and then you selectively absolve bad performance metrics for anyone you don't want fired.

Of course, one also has to be careful in this situation and not have employees discuss their performance ratings, and especially don't have anyone else in management discuss performance rating absolutions. Think of the horrible legal s

First, if a system is that good, then managers won't be able to game it to play favorites.

This is the most likely reason I see why no performance metric that works would actually be picked up. The ideal rating system for a corrupt manager is one where everyone is rated poorly, and then...

An ugly world (I know it from my past experience)... I mean the world in which a manager doesn't compete with other managers (on budgets, prestige, etc) based on successes, but is "chosen to be a manager because we need one and... let's see who's the most notable and less-covered-by-shit on the floor".In the last case, the chosen person will be the most-full-of-shit-and-willing-to-spread it one... you can imagine how the managed team stinks.

The first big problem though is lack of metrics and insight into what developers are doing. I worked on a team and helped implement big brother software on a large development team's computers. The problem was that there was too much data and the company that implemented the software then tried to provide inadequate visualization (e.g. 40% of time was spent on JSP/Java editing, 10% on a web browser, etc.) and just knowing *what* people are working on is not enough. You have to have metrics based on the outp

On a recent project there were two developerss fixing bugs. "A" would fix a bug in 4 hours rewriting 50 lines of code. "B" would fix the a bug in 9 hours rewriting 1 line of code. They let B go because A was more productive. Funny thing is they never ran out of bugs.

If you can't trust your first level managers to understand who is contributing w/o a pile of metrics, there is a simple solution -- get new managers.

ICs will game metrics much more than managers. Good managers rarely have a vested interest in favoring one developer over another except to the extent that one developer contributes more than another to getting reliable systems meeting requirements to the customer.

If you have metrics that "managers can't game", it means that ICs can game them because they

Wouldn't it be the project leader who monitors these on an individual basis? If a coder isn't pulling their weight its up to the project leader to address it up to the point of termination. Above that you have a suit who monitors the project leader's team performance and decides how well the project leader is doing. Of all the places layered management doesn't work, coding is not one of them. It's a challenge to hold a developer accountable because there are so many different approaches to the same problem in coding and a lot have definitive pros and cons.

No. The - for the sake of politeness, let's call them "people" - who invested time and effort into devising these schemes have actually built a complete chain of negative productivity by doing so. Remarkable.

Personally I am always happy with the guy who can get things done with one line of code instead of a hundred, but what I really care about is that objective is met and we don't have a host of bugs that require 10 times the cost of the development just to maintain. Its not hard stuff but it does require common sense and a hard nosed attitude both of which can be scarce commodities these days.

It isn't that cut and clear. I can write a bug-free one line perl solution, but what happens when someone needs to maintain it? Objectives come from all directions. Your shop wants quick turn around and maximum profit. The customer wants quick turn around , solid code, AND a cheap price. Your goal is to be productive and get home before dinner. Notice the conflicting objectives.

I don't know how many project charters have come across my desk that have the project goal as "To create software that..."

I work at a food company. A multibillion dollar food company. We do food research. We sell food. We do NOT turn a profit writting software. So no project should ever come to me with a goal of creating software.

You are correct in that the only metric that matters to the business is the profit. But in order to determine individual contribution, you can't just lo

Personally I am always happy with the guy who can get things done with one line of code instead of a hundred, but what I really care about is that objective is met and we don't have a host of bugs that require 10 times the cost of the development just to maintain. Its not hard stuff but it does require common sense and a hard nosed attitude both of which can be scarce commodities these days.

I am also REALLY happy to have "that guy" that has absolutely shit productivity, but somehow manages to pick up on every time a "solution" is proposed by the rest of the team to a problem that doesn't exist or doesn't matter, and stops THEM from being really efficient at doing the wrong thing. I'm also really happy to have "that girl" that doesn't seem to really be doing anything, but take her out of the team and everyone else starts floundering because she's actually constantly helping them be a lot more productive.
"Meeting the objective" is actually potentially just as bad as any other metric, because it depends on how you define the objective, and meeting it. What the customer asked? What the customer wanted? Or what the customer actually needed?

Personally I am always happy with the guy who can get things done with one line of code instead of a hundred, but what I really care about is that objective is met and we don't have a host of bugs that require 10 times the cost of the development just to maintain. Its not hard stuff but it does require common sense and a hard nosed attitude both of which can be scarce commodities these days.

I am also REALLY happy to have "that guy" that has absolutely shit productivity, but somehow manages to pick up on every time a "solution" is proposed by the rest of the team to a problem that doesn't exist or doesn't matter, and stops THEM from being really efficient at doing the wrong thing.
I'm also really happy to have "that girl" that doesn't seem to really be doing anything, but take her out of the team and everyone else starts floundering because she's actually constantly helping them be a lot more productive.

"Meeting the objective" is actually potentially just as bad as any other metric, because it depends on how you define the objective, and meeting it. What the customer asked? What the customer wanted? Or what the customer actually needed?

That guy and that girl are generally called project/team leaders. Don't fret, You raised an important issue. The guy you don't want on the team is the one that comes up with ridiculous edge cases and is needlessly obtuse. Like someone who invents coders that are doing the work of managers and senior managers then complains a measurement tool doesn't capture their contribution, or goes into a recursive loop trying to figure out what the objective should be.

If they know their productivity is being measured, it becomes a contest to see who can cook the books the best anyway.

Yes. And this would be a Good Thing if the metrics actually measured the right thing. The problem is, the metrics represent something that isn't actually what you want, so developers are motivated to do something that isn't actually what you want.

So if you are going to be using a metric at all, the first questions you should ask yourself is what the metric measures, and if that is actually what you care about. For example, you could measure how many lines of code people produce, and reward people based on t

By analogy with metadata - data about data - "metametrics" was what sprang to my mind. And like most things that do after I've had a few jars, it a) already exists [mac.com] and b) is far too shit to deserve such a goshdarn clever name.

What about all the time you spend not coding: meetings, documentation, training users on how to use the system, working out the design before you start coding, answering emails, sending status reports, filling out time sheets, coordinating work with other developers, coordinating things with others in IT so your program will have a server to run from, being the go-between for the IT server team and the customer when the server goes down, creating database layouts and writing SQL?

Not according to some of the developers around here. Of course, these are the types that put "select * from table" into their Java code, then try to filter data in the application layer. Then they whine that their application is slow.

I write SQL most of the day, and in my opinion, it's more difficult to do well than traditional programming. If I'm writing a webapp somewhere, I only have to keep the individual logic i'm working on in my head - with SQL, I may have to know how a single dataset is processed over a half dozen steps to get the result I want.

This all comes down to lazy, gutless management. Why take the time to get to know Dave and monitor the quality of Dave's work when we can just look at a spreadsheet at the end of the month? Managers prefer to tinker with automatic analysis software rather than manage.

Which is more fun, getting a better handle on what Dave is doing, or researching fancy new software tools that might get you all sorts of praise from metric-craving executives?

Dave's job, which was once about creating a quality product, now shifts to merely satisfying the metrics.

I hope it's different where you work, but where I work, the management is lucky if they are truly 100% management. But in reality, the managers of technical teams also have to play the role of project manager, technical lead, and sometimes architect. Yes, it sucks, and I wish it weren't that way, but trying to change that is like pulling teeth.

So even if Mr. Manager has good intentions, he really is prevented by other duties from doing his job to the fullest extent. It sucks, but I wouldn't feel right

What applies to Dave applies to Bill, Jane, etc just as much. Now they're all more interested in looking good on the metric of the day than on producing a decent product. Any who make the mistake of focusing on the product will be eliminated systematically by the metric.

Metrics are valuable if you do the same thing repeatedly. If you build a new building that is like the previous one, you can collect metrics and compare your performance against history. If you write the same search algorithm again and again, you can collect metrics and compare to see how your performance changes over time.
Of course, with software, you never repeat. Somewhere around the third time, you move it into some form of library, reuse it, and start on a fresh problem.
Perhaps metrics are helpful in some situations, such if your team keeps repeating the same mistakes, you might find similarity in those mistakes (code smells.) There are plenty of people working on these problems and tools.
But, from a management point of view, if you keep doing the same thing, you are doing it wrong, and code metrics are not going to help much.

I don't get the concept of everything needing to be quantified. Does the team accomplish what the goals of having the team are? Does it get developed in a fair timeframe? Is everyone on the team pulling their own weight, or are there complaints of someone slacking off? In the end if the product works then the team is doing well, if the product isn't there should be at least one hybrid manager/coder that actually works with the team members sees who is committing what and can tell off the bat if there is or isn't a weak link dragging the rest down. Actually putting a pen and paper number on a complex project is silly. Do authors get judged by the number of pages they write in a day, no they get paid by the success or failure of the book. You can't judge by the number of lines of code, bugs per line ratio or anything like that, because it is all subjective and has little to no bearing on the end product.

... Do authors get judged by the number of pages they write in a day, no they get paid by the success or failure of the book. You can't judge by the number of lines of code, bugs per line ratio or anything like that, because it is all subjective and has little to no bearing on the end product.

Many successful novels, started out as magazine serials, which were paid by the word. Usually, though, they are edited before they are put into book form.

I've seen a bunch of movies, where I thought, "This would be a great movie, if they just cut half of it, then sped the result up by 1.5x"

I've often thought that coders should be paid by how *little* they write, as long as it does the job.

Measuring developer output/metrics effectively is a tough problem. Developers could solve it, but if they do, then they have to both change the way that they work and possibly work harder. Developers are smart enough to know that the metrics will be misused, even if the logic used to produce them is valid. Therefore, any solution will be ridiculed by the development community as insufficient, but the degree to which it is ridiculed will lessen as the solution improves. A solution though, is inevitable if development continues.

That's what I said, but you said it more concisely. If this were code, your solution would probably have rated higher than mine.
But, after many (too many) years as a developer now, I can say that programming is about as much an art as carpentry. You can do beautiful things with both, but in the end it is about the artifacts and how they are used. They *are* measurable, and providing decent metrics for software development *is* possible. The biggest problem is that we don't want to admit that it is just a

I've generally assumed that standardized evaluation is pushed by upper management to provide a baseline for the direct mangers to follow. Sure, a manager should know which of his direct reports are capable and which aren't, but how can you compare one manager's "adequate" to another's? A few numbers won't get you a complete picture, but combine those numbers with the manager's opinions and you can better calibrate performance than you could with either evaluation on its own. A PHB will be a PHB no matter wh

Yeah, that's an upper management failure in my book. They should be connected well enough to know what is going on at least two levels down, not one (enabling them to check the calibration of their direct reports). If that's not possible, the branching factor is probably too high.

I think the idea of "productivity" is a hold-over of the Industrial Revolution that does not pertain to many of today's jobs; jobs where the unit of work is hard to define, and ultimately irrelevant. Are you telling me you pick your doctor by how many patients he can see in a day? Probably quite the opposite!

In terms of software development, I find that the *effectiveness* of a developer is more important, where effectiveness considers the following (not an exhaustive list):
- Appropriateness of solution
- Thoroughness of implementation (logging, exception handling, graceful failure, input validation, etc.)
- Well-written, parsimonious code that is easy to read and descriptive of what it does
- Works right the first time, no kickbacks from QA or end user

Give me someone who is effective but slow over someone who craps out junk quickly any day of the week and twice on Sunday! In the end, I don't care about productivity metrics, I care that the end users get a useful piece of software that does what they need with a minimum of headaches.

Instead of trying to micro manage your devs ("Oh, Frank, you created 10 lines less than Bob, you slacker!"), focus on the project instead. Check whether your teams meet their deadlines. Pit the teams against each other. And rely on peer pressure within teams to keep the productivity up and slackers from slacking. Are you judging your beancounters by the amount of bookkeeping lines they fill out per day? Do you gauge the productivity of your project managers by the amount of meetings they go to? No, you meas

No doubt you can measure quality of code or the competency of a programmer. But can you stablish an objective metric with fast results (remember, corportations are "this quarter" oriented) that gets it right?

And every developer starts as an engineer, and ends as an artist. As simple as that.
The engineer can do anything, but have not done it yet.
The artist knows what to NOT DO, because he has already done it, and does not wanna to repeat all the teenage mistakes again.
The artist just gives you the solution. End of story.