Lean Metrics for Agile Software Configuration Management

As mentioned earlier, measuring the overall cycle-times for each of these types/levels of build, as well as the overall cycle-time to progress through all the build types/levels from beginning to end, can be important measures for an Agile CM environment. Drilling down further to identify waste/bottlenecks could entail measuring the segments of each of those cycle-times that correspond to waiting, information-seeking, decision-making, and integration/build server processing time and utilization.

In the arena of change/request management, harmonic cadences would also address release cycle planning for a product-line of products that are releases of multiple (internal and external) component releases. One can think of an overall portfolio/project and governance/CCB strategy as encompassing progressively larger levels of scale: requirement, change, use-case, feature, iteration, project/release, product/program, etc. ITIL service management addresses the various levels of these across an enterprise. Nested synchronization and harmonic cadences address how to integrate the management of networks of queues and sub-queues diverging and converging across all impacted elements of an enterprise architecture and its applications, services, and the products the enterprise uses those to produce.

Nested synchronization would seem to apply to branching structures where development branches feed into integration/release branches and their relation to mainline branches, and the direction and frequency with which changes get merged or propagated across codelines. Here again the cycle-time for "transferring" changes from one branch to the next (as well as through all the branching levels) can be a useful measurement for ensuring "flow."

Of course, when you can manage without the "nesting", that is ideal for continuous integration. Continuous integration together with test-driven development seems to approximate what Lean calls one piece flow. An article from Strategos discusses when one-piece flow is and isn't applicable.

In the context of SCM, particularly continuous integration and TDD, one piece flow would correspond to developing the smallest possible testable behavior, then integrating it once it is working, and then doing the next elementary "piece", and so on. This is typically bounded by:

the time it takes to [correctly] code the test and the behavior

the time it takes to synchronize (merge) your workspace (sandbox) with the codeline prior to building & testing it, and ...

the time it takes to verify (build & test) the result

So measuring each of the above (rather fine-grained) could also prove useful -- provided that it could be done unobtrusively. If one has instituted and automated a repeatable process for creating reproducible builds, then the measures can often be performed or triggered automatically during the automated process (or the data can be generated and logged for subsequent collection and analysis).

Note that working in such extremely fine-grained increments might not always work well if the one-piece-flow cycle-time was dominated by the time to synchronize (e.g., rebase), or to build & test, or if it usually resulted in a substantially disruptive or destabilizing effect on the codeline. In those cases, if the time/cost "hit" was more or less the same (independent of the size/duration of the change), since the penalty per "batch" is roughly the same for a batch-size of one piece as it is for a larger batch-size, then it makes sense to develop (and "transfer") in larger increments ("batch-sizes") before integrating and committing changes to the codeline. Hence making some basic measurements to monitor these values can give an indication of when to stop striving for one-piece flow, or else when to revisit the build process to eliminate bottlenecks!

Experiences with Version Control-related Metrics We've seen one or more of the following being gathered (not necessarily all of them on the same project):

mean & median commit frequency

average commit "granularity" (delta-lines-of-code per commit)

average number of changed files per commit

average number of lines changed/added/removed per file changed per commit

average number of files with merge-conflicts per commit

average time to resolve merge-conflicts per commit

average build-time for a private build (usually an incremental build)

build-time for an integration build

average integration-test cycle-time (time to run all the tests)

change volatility report (a report of the most frequently modified files over a given time period)

change collision report (a report of files that had the most parallel development and/or checkout-lock-contention during a given time)

Granted, the last two items above aren't really metrics (they're reports), but hopefully you get the idea.

Metrics regarding change-tasks' size and commit-frequency are relative rather than absolute metrics. We want to observe the trend over time so we might notice when: integration frequency went from ~2hrs last iteration to ~3hrs this iteration while the change-task size stayed mostly the same; or maybe the task-size increased as well, in which case we might ask if we were perhaps integrating too frequently before, or are we integrating to infrequently now, and why?

Build-time and test-time are important by themselves as absolute numbers because they help give us a lower-bound on the minimum time required to commit changes. And might help us decide whether we should try to do full builds instead of incremental, just as an example.

Combine all of those above and they basically give us an idea of approximately how many changes might be committed (or be prevented from being serially committed) during the time it takes to build & test. This might help us decide when/if to switch from single-threaded commits to multi-threaded or vice-versa, or whether to go from running partial builds/tests to more complete builds/tests.

The information about number of conflicts and time to resolves them can also help feed into the above, plus also gives as some "trends" about how disjoint are change-tasks typically are. If there is an increasing trend of parallelism, then maybe we need to ask if we are doing a good job of defining the boundaries of a use-case (story), or of splitting up the work for story effectively.

About the author

Robert Cowham

Robert Cowham has long been interested in software configuration management while retaining the attitude of a generalist with experience and skills in many aspects of software development. A regular presenter at conferences, he authored the Agile SCM column within the CM Journal together with Brad Appleton and Steve Berczuk. His day job is as Services Director for Square Mile Systems whose main focus is on skills and techniques for infrastructure configuration management and DCIM (Data Center Infrastructure Management) - applying configuration management principles to hardware documentation and implementation as well as mapping ITIL services to the underlying layers.