Featured in
Architecture & Design

Mini-talks: The Machine Intelligence Landscape: A Venture Capital Perspective by David Beyer. The future of global, trustless transactions on the largest graph: blockchain by Olaf Carlson-Wee. Algorithms for Anti-Money Laundering by Richard Minerich.

Featured in
Operations & Infrastructure

Mini-talks: The Machine Intelligence Landscape: A Venture Capital Perspective by David Beyer. The future of global, trustless transactions on the largest graph: blockchain by Olaf Carlson-Wee. Algorithms for Anti-Money Laundering by Richard Minerich.

Featured in
Enterprise Architecture

Mini-talks: The Machine Intelligence Landscape: A Venture Capital Perspective by David Beyer. The future of global, trustless transactions on the largest graph: blockchain by Olaf Carlson-Wee. Algorithms for Anti-Money Laundering by Richard Minerich.

IT Projects: 400% Over-Budget and only 25% of Benefits Realized

An alarming study by Flyvbjerg and Budzier published in the Harvard Business Review has made everyone stand-up and take notice. The coherent advice being that IT projects are much more riskier than we think.

The study showed that amongst the 1400+ projects surveyed, on an average 27% were over budget. 16% of the projects could easily be categorized as Black Swans. They had a cost overrun of over 200% and were late by almost 70%. These were the projects which could cause stunning failures and bring down companies.

These results are significant. But even more frightening are the conclusions of the authors of Harvard Business Review article. They write: If you want to avoid the slow death caused by IT projects you must be prepared not only to spend 400% more than planned on the project, but to reap only 25% of the expected benefits. If you keep this in mind you can possibly prevent a company from being killed by an IT project.

My theory is that it is these very large re-platforming projects that are frequently the issue. They touch so many systems, so many discrete business units involved across the globe, and the desire to do customization so they become impossible to upgrade.

Ed mentioned a similar re-platforming project which was successful as it had all the ingredients of a good Agile project.

An example from the article of a successful project by the Emirates bank in 2006, shows a successful way to run a large complex project. The lessons learned from this successful project were to:
1. Stick to schedule
2. Resist changes to the projects scope
3. Break the project into discrete modules
4. Assemble the right team
5. Prevent turnover among team members
6. Frame the initiative as a business endeavor, not a technical one
7. Focus on a single target – readiness to go live, measuring every activity against it.

But maybe we are addressing the wrong people , like Ken Schwaber says in his new, soon to be published book. Possibly IT customers, software development, and product development need to simply not accept any more that IT departments, QA departments and process people can not create finished, tangible and measurable results every two weeks.

Real governance means leadership. Real governance means courage-the courage to tell the truth even when it will not be well-received. Too many "well"-run organizations operate under a culture of fear and "group-think". This is why IT projects or any projects for that matter, have the ability to bring an organization to its knees. We have to work on building organizational cultures that welcome honesty, diverse opinions, and integrative thinking.

Thus, one could dive deeper to analyze the shortcomings and learn from them, however the numbers do suggest a serious re-think in the way projects are executed.

I personally believe that the root of the problem is called out in the article but in a subtle way (probably in order to avoid offending their readers). The issue is that in most non-software companies, information technology is treated like something separate from the business. The conventional wisdom of a decade ago dominates. That is, if your company is not producing software for sale, it shouldn't be building software. It should be purchased, like Word or Excel.

This line of reasoning seems pretty bullet-proof until you dig into the details and it leads to a number of problems including the issues in the article.

First, lets be clear, no company should seek to build software for internal use that can be purchased at reasonable cost and can can meet their needs without customization. Secondly, companies should seek to purchase or other wise acquire software that implements the generic functions and allows for powerful customizations.

The issue arises when companies purchase something like SAP assuming that their needs are sufficiently supported by the software and that anything they need that is special will be supported by 'configuration'. What then happens is that the company realizes (after investing large amounts of money) that they can either make their business completely generic and lose all strategic advantage over competitors or they can pile on a lot more money to customize the hell out of it. Given the options, most high-level execs prefer more investment over losing strategic advantage. Some companies (I suppose) choose the other option and a lot of IT people think this is a good idea (it isn't.) You might be more likely to succeed at the project but the company will often be ruined in the process.

The error really is taking on a project based on false assumptions. Once that mistake is made, there are no good options. Companies need to stop thinking that information technology is separate from their core functions. Most information technology is only worth having if it is tightly coupled to the design of the business and often technology imposes fundamental constraints on business.

No project management methodologies or helpful tips will address this problem. Companies need to embrace technology and make IT a stakeholder at the highest levels of the organization for the kinds of problems described in the article to be fully addressed.

1) Seriously, who does large IT projects anymore? Did they not get the memo that they are, at best, doomed to be way over budget and behind schedule? At worst, they are an abject and complete failure. Hell, the article starts off with a story of Levi Strauss bringing in Deloitte consultants. Even without knowing what they were working on, I could have told you it would fail right there.

2) Black Swans are unpredictable. What's happening on these projects could be seen from a mile away. I guarantee you that any competent developer with 10+ years experience under their belt could have told you these would fail long before they did (and often, before they even started).

The take away shouldn't be that large projects are risky. It should be that you just don't do them in the first place. They have so many problems with them that aren't even technical that it's pointless to even begin them. I've seen companies that always execute small projects well try a big project and have it be a huge failure.

Dean, you are right that the sample list might seem skewed but this is what they follow it up with (from the article), which makes it relevant for all of us

Our sample drew heavily on public agencies (92%) and U.S.-based projects (83%), but we found little difference between them and projects at the government agencies, private companies, and European organizations that made up the rest of our sample.

True Black Swans are unpredictable however in the hindsight, they do not seem improbable. Like they described the Nike project ..

A $5 million project that leads to an almost $200 million loss is a classic “black swan.” The term was coined by our colleague Nassim Nicholas Taleb to describe high-impact events that are rare and unpredictable but in retrospect seem not so improbable.

Why is a project failure if it doesn't live up to the initial plan? It's more likely that the initial plans were wrong. Why do we assume we can predict such complex projects with so many unknowns? Please pass me the crystal ball.

True Black Swans are unpredictable however in the hindsight, they do not seem improbable. Like they described the Nike project ..

I think that Marc's point is that a lot of people could (and probably did) predict these issues long before they occurred. If something occurs as frequently as large project failures do, calling it a 'black swan' doesn't seem inappropriate.

Aside from my comments above, the dynamic of such projects is that middle management cannot openly express doubts for fear of demoralizing the team. It's not that the failures are improbable based on history (quite the opposite) but that the people involved are simply unaware of the risks.

Very interesting stuff. Seems large-scale IT is in even worse shape than indicated by the past studies people have cited over the years. Unclear why the item is categorized under the heading, "agile," though. I see no connection.

I got the hbr article, read it and did some google queries. It turns out that Flyvbjerg is a consultant that supports the "reference class forecasting" position of his firm. His firm specializes in a lot of Civil and Construction projects. Nothing wrong with that of course, but reference class forecasting( finding projects like yours and then basing your budgeting on it) is a challenge. Since every company is different, and since software is so behavioral, I doubt that this forecasting method would do much for us. Remember, this is a complex system, and therefore response better to adaptive than control based process models (google Stacey Diagram: Zone of complexity). The whole concept of the Black Swan that they throw around in the article is about the impossibility of prediction.

The background of the fellows in this article explains why the 7 key steps don't resonant completely with me and others on this list. I've been on "stick to the schedule - limit scope" projects, and they were called waterfall. They release something ontime that no one wants that people who use it haven't seen, and who are usually less productive afterwards. In the article he talks about how Emirates went "Big Bang" and how great that was. Call me skeptical. If Scrum is right, then we use adaptive processes for those things, and we learn and adapt towards deployment.

However, the last 5 are very good, especially #7, focusing on a single target. Focus is what is missing from my enterprises that have had tricky large scale projects.

Number two may strike us as an Agile faux pas, not allowing change in scope, but I believe the authors are referring to “Mission Creep” rather than changing featues in a requirements spec. Drifting missions or just unclear goals strikes me as a hallmark of any unsuccessful endeavor.