January 08, 2009

Metriculation - Metrics that Lie

Metrics can be a valuable asset to software development teams in
helping to improve their efficiences and measure their effectiveness.
Unfortunately, metrics don't always provide the intended value.
Sometimes, metrics lie.

Metriculation is a term I use to describe how metrics are made to lie.
It's derived from statisticulation, a term introduced by Darrell Huff
in his book, "How to Lie with Statistics", first published way back in
1954. There are many forms of metriculation, and it's a significant
factor that affects the successful use of software development
metrics.

Metriculation is not the same as using metrics
ineffectively, or measuring the wrong thing. For instance, using LOC
(lines of code) was once thought an effective way to assess programmer productivity.
LOC has nothing to do with productivity. Code generation wizards can be
used to create a component with thousands of lines of code, but it
might take days to correct a nasty bug that requires changing only a
single LOC. LOC does have it's value, however. Using LOC to help
calculate the percentage of code under test, or to help with software
estimation, are examples. Using LOC to gauge developer productivity is
ineffective, but it's not metriculation.

Instead, metriculation involves using metrics to mislead. Downright
deceive. Lie! For instance, in an effort to undermine the value of unit
testing, a developer may claim that 75% of the system defects are found
in code with unit tests. This form of metriculation, known as
semi-attachment, is done by finding a way to prove what you want to
prove by drawing false conclusions. Based on only this information, one
might be drawn to the false conclusion that unit testing does not
contribute to higher quality software. Unfortunately, what the
developer may not have shared is that 95% of the code is under test,
meaning that the remaining 25% of defects reside in 5% of the code.
Given this additional information, we could have arrived at an entirely
different conclusion - that unit testing does contribute to higher code
quality. In this example, the metriculation was intentional, but that's
not always the case. Often times, it's due to oversight. Regardless,
metriculation is dangerous.

In addition to semi-attachment, there are many other forms of
metriculation, all with numerous real world examples of how they've
been used to push an agenda. These include selective use, biased
samples, well-chosen average, fudging the numbers, overgeneralization,
faulty assumptions, whiz bang graphics, and creative charts. Upcoming
Burton Group research will discuss in more detail metriculation, and
how we can deal with. As that material matures, I'll continue to share
interesting snippets.