All of these factors are commonplace in the software development process. A recent Standish Group report indicates that only 35 percent of software projects are successful, meaning they met the deadline within budget and satisfied client needs. Some 19 percent were outright failures, with the rest muddling through in the middle ground.

Which begs the question: Why? Why is software development, which is done by groups of intelligent professionals, so god-awful difficult?

Scott Rosenberg, author of Dreaming in Code, a book about the development process, has mulled this question extensively. Having interviewed legions of programmers and been involved with software projects himself, hes come to some conclusions.

"Dreaming in Code" by Scott Rosenberg

First, though development is a profoundly technical process, its not the technology that creates snafus. Rather, its the human-technology interface. Specifically, The difficulty that people in management positions have in gaining visibility into the technical world. And also  and more worrisome  The assumptions that people make about software.

For instance, compare software development with the process of building a bridge. The mix of steel and concrete that goes into bridge building is tangible to the casual observer. We can look at it and say, Its about halfway done.

But with software? How are upper level execs supposed to know that a mass of code is halfway done?

They could, of course, ask the lead programmer, and in fact they often do. But thats where the problems usually start.

In the mind of the lead programmer, once the project is
decomposed into a series of technical problems, he can say with some assurance, Oh, this module is halfway done, or this module, we havent started on that. Those questions can be answered.

But when that programmer reports his progress to management, hes talking to people who dont see things as he does.

Rosenberg notes: The assumption is, because we know its so easy to change  its code, right? We didnt pour concrete into a foundation  that its not difficult to change things.

Furthermore, Programmers are often, to their credit, eager to please, and eager to prove their versatility and their ability to nimble. And so we dont apply the same restraint and self-discipline when building software that we tend to use in physical world projects. We get a quarter of the way or halfway into it, and say, Lets add some things.

The innocent phrase Lets add some things is rarely beneficial to the sanity of the programmers.

In fairness to management (a group that might not deserve such fairness), sometimes the very act of development opens new possibilities. So perhaps its only right and natural that they shift course in midstream.

And maybe as its being built they get a clearer picture, and suddenly they see something they didnt see before. Then they go, Oh, I guess we should rebuild this part, and they go back to the engineers and say, You know, we didnt realize this.

Although its rarely the programmers who turn projects into quicksand, sometimes the classic programmers personality profile comes into play.

Theres often a particular personality type that goes into the field, Rosenberg says. To master the technical complexity of getting the computer to do exactly what you want it to do, you have to be very precise. You have to be able to think in a way that makes that easy for you.

Sometimes  not always, but sometimes  if you are the kind of person who is able to think really easily about how to get a computer to do what you want, youre not always the most imaginative person to think about how users are going to do something. This is a classic issue in the field.

Users (or at least certain users) are remarkably non-technical. Simple computer tasks confuse them terribly. For a technologically sophisticated programmer to put himself into the mindset of a blubbering newbie is difficult.