I consult, write, and speak on running better technology businesses (tech firms and IT captives) and the things that make it possible: good governance behaviors (activist investing in IT), what matters most (results, not effort), how we organize (restructure from the technologically abstract to the business concrete), how we execute and manage (replacing industrial with professional), how we plan (debunking the myth of control), and how we pay the bills (capital-intensive financing and budgeting in an agile world). I am increasingly interested in robustness over optimization.

I work for ThoughtWorks, the global leader in software delivery and consulting.

Monday, April 28, 2008

In the wake of a credit market seizure, illiquid investments, $245 billion of write-downs and losses1, collapsing funds and financial institutions, and no indication as to where it’s going to end, US capital markets are facing significant changes in how they're regulated. Hedge funds are a flashpoint. There are about 8,000 funds managing some $2 trillion of assets,2 and there is no way of knowing whether or not there’s a large write-down looming somewhere among them. Indeterminate counterparty risk in a highly interconnected financial system means there’s a chance capital markets could get blindsided yet again, so hedge funds are front and centre of the regulatory debate.

There are two schools of thought over how hedge funds should be regulated.

Members of Congress are calling for strict, rule-based regulation. Very few industries have a track record of successful self-regulation, and capital markets firms have incurred more than a few self-inflicted wounds of late. Rule-based regulation calls for tight controls on activity. Transparency is an assumed byproduct: if actions are pre-defined, everybody will know exactly what everybody else is up to. There is also an “I pay, I say” dimension: if the US taxpayer could end up footing the bill, the taxpayer must have the opportunity to set the rules. The champions of rule-based regulation believe this is accomplished through control and regulation, imposed through legislation and agency.

The US Treasury department is agitating for principles to play a greater role in regulation. Because capital is globally mobile, markets must innovate to remain competitive. Financial markets are innovating at a fast clip. Rules can't be written as quickly as markets evolve. Principle-based regulation posits that compliance with best practices is the best way to facilitate innovation while retaining transparency. Advocates of principle-based regulation argue that it is in everybody’s best interests to voluntarily comply, as compliance guarantees consistency – and with it transparency, liquidity and confidence – in capital markets.

This debate mirrors a similar phenomenon in IT.

The traditional approach to IT project management is consistent with “regulation by rule.” This camp values practices such as deterministic project plans, highly detailed task orders, explicit role definitions, and timesheet-based project tracking. The theory is that consistency is achieved through meticulous control; any deviation from plan is visible and immediately correctable. At the other extreme are the Agilists who champion regulation through principle. This camp values practices such as test driven design, continuous integration, co-located and cross-functional teams, short development iterations, and frequent releases of software. They argue that innovation, transparency, consistency and ultimately project success result from compliance with best practices more than they are adherence to a collection of rules.

Not surprisingly, the ideological arguments in IT are similar to their capital markets counterparts. Those who advocate the traditional approach argue that top-down control is essential, and that best practices are ignored by teams when things are going well. How can there be self-regulation in an industry notorious for significant overruns and spectacular project failures? Why would a business abdicate responsibility for oversight if there's a risk it will have to bail out a project? The Agilists argue that top-down control is a myth, and that everybody has a vested interest in adhering to best practices. How can anybody expect that deterministic project planning will keep pace with changes and discoveries made during development? And how can we expect innovation in an environment stifled by bureaucratic control systems that are not aligned with day-to-day execution?

“Control” is elusive in IT, particularly at the high end. Applications with the potential to yield significant business impact typically involve new processes or technologies. In these cases, development is an exercise of continuous problem solving, not rote execution. It isn’t practical to create deterministic project plans for the delivery of solutions not yet formed to problems not yet discovered. Additionally, history has shown that regulation and control do not offer deliverance from failure, let alone disaster. As US Treasury Secretary Henry Paulson commented in the aftermath of the Bear Sterns intervention, “I think it was surprising … that where we had some of the biggest issues in capital markets were with the regulated financial institutions.”3 The same can be said about IT. Rules offer no guarantee of effective risk management, as time and again, we have seen delays or functional mis-fits announced late in the lifecycle of even the most tightly “controlled” IT project.

If IT is to be a source of innovation and business responsiveness, it needs disciplined execution more than it needs imposed rules. Unfortunately, “disciplined execution” doesn’t describe how the vast majority of IT is practiced today. IT has launched its share of self-targeted missiles over the years, and its track record remains poor. On top of it, buying patterns increasingly relegate IT to utility status; they don't elevate it to strategic partnership. Principle-based regulation may be appropriate for IT, but it faces significant headwinds.

This debate will affect the role and relevancy of IT in the coming years. There is an opportunity for IT to take leadership in this debate, but it can do so only if it has its house in order. Without principled execution, IT will increasingly be treated as a utility, regulated by rule. But by adhering to best practices, IT can demonstrate an ability to self-regulate. This will allow IT to strike a balance between effective practices and the rules with which it must comply, and position itself to be a driver of alpha returns.