Sunday, May 4, 2008

An Under-Appreciated Fact: We Don't Know How We Program

I was talking to a colleague from another part of the company a couple of weeks ago, and I mentioned the famous ten-to-one productivity variation between the best and worst programmers. He was surprised, so I sketched some graphs and added a few anecdotes. He then proposed a simple solution: "Obviously the programmers at the bottom end are using the wrong process, so send them on a course to teach them the right process."

My immediate response, I freely admit, was to open and shut my mouth a couple of times while trying to think of response more diplomatic than "How could anyone be so dumb as to suggest that?". But I have been mulling over that conversation, and I have come to the conclusion that the suggestion was not dumb at all. The problem lies not with my colleague's intelligence but in a simple fact. It is so basic that nobody in the software industry notices it, but nobody outside the industry knows it. The fact is this: there is no process for programming.

Software development abounds with processes of course: we have processes for requirements engineering, requirements management, configuration management, design review, code review, test design, test review, and on and on. Massive process documents are written. Huge diagrams are drawn with dozens of boxes to try to encompass the complexity of the process, and still they are gross oversimplifications of what needs to happen. And yet in every one of these processes and diagrams there is a box which basically says "write the code", and ought to be subtitled "(and here a miracle occurs)". Because the process underneath that box is very simple: read the problem, think hard until a solution occurs to you, and then write down the solution. That is all we really know about it.

To anyone who has written a significant piece of software this fact is so obvious that it seems to go without saying. We were taught to program by having small examples of code explained to us, and then we practiced producing similar examples. Over time the examples got larger and the concepts behind them more esoteric. Loops and arrays were introduced, then pointers, lists, trees, recursion, all the things you have to know to be a competent programmer. Like many developers I took a 3 year degree course in this stuff. But at no point during those three years did any lecturer actually tell me how to program. Like everyone else, I absorbed it through osmosis.

But to anyone outside the software world this seems very strange. Think about other important areas of human endeavor: driving a car, flying a plane, running a company, designing a house, teaching a child, curing a disease, selling insurance, fighting a lawsuit. In every case the core of the activity is well understood: it is written down, taught and learned. The process of learning the activity is repeatable: if you apply yourself sufficiently then you will get it. Aptitude consists mostly of having sufficient memory capacity and mental speed to learn the material and then execute it efficiently and reliably. Of course in all these fields there are differences in ability that transcend the mere application of process. But basic competence is generally within reach of anyone with a good memory and average mental agility. It is also true that motor skills such as swimming or steering a car take practice rather than book learning, but programming does not require any of those.

People outside the software industry assume, quite reasonably, that software is just like all the other professional skills; that we take a body of knowledge and apply it systematically to particular circumstances. It follows that variation in productivity and quality is a solvable problem, and that the solution lies in imposing uniformity. If a project is behind schedule then people need to be encouraged to crank through the process longer and faster. If quality is poor then either the process is defective or people are not following it properly. All of this is part of the job of process improvement, which is itself a professional skill that consists of systematically applying a body of knowledge to particular circumstances.

But if there is no process then you can't improve it. The whole machinery of process improvement loses traction and flails at thin air, like Wiley Coyote running off a cliff. So the next time someone in your organisation says something seemingly dumb about software process improvement, try explaining that software engineering has processes for everything except actually writing software.

Update: Some of the discussion here, and on Reddit and Hacker News is arguing that many other important activities are creative, such as architecture and graphic design. Note that I didn't actually mention "architecture" as a profession, I said "designing a house" (i.e. the next McMansion on the subdivision, not one of Frank Lloyd Wright's creations). People give architects and graphic designers room to be creative because social convention declares that their work needs it. The problem for software is that non-software-developers don't see anything creative about it.

The point of this post is not that software "ought" to be more creative or that architecture "ought" to be less. The point is that we need to change our rhetoric when explaining the problem. Declaring software to be creative looks to the rest of the world like a sort of "art envy", or else special pleading to be let off the hook for project overruns and unreliable software. Emphasising the lack of a foundational process helps demonstrate that software really does have something in common with the "creative" activities.

Actually I think programming is 100% design. The source code is just the final representation of that design.

But I disagree with your last paragraph: when you look at design tasks in other professions you do find processes. Take a look around the next office building you are in. They all have the same basic requirements, and this leads to the same basic designs, which is why they all feel basically the same.

The thought process in programming looks alien to most people in business, but there is a group to whom it would be perfectly natural: mathematicians. They basically just have the problem, and maybe some interesting properties of the problem and similar problems, but there is no process in the world that could ever produce a sound proof for any given problem, but sit a team of mathematicians down with it and sooner or later a solution will be produced.

This ties in very intimately to Fred Brooks' thought. The difference between software (or maths) and architecture is that once a problem is solved, it should never be solved again. (Except for didactic purposes, or finding a more general/elegant solution, but the point still stands). Wheras, for architects, they basically build the same buildings using the same components over and over again, which is exactly where processes shine.

I agree that there is no process for writing a program, and that code is all design. But I think you're being unfair on real architecture.

Sure, many modern office buildings all look the same, but if you read Christopher Alexander's (the father of design patterns) original work on real architecture you'll see that shouldn't be the case.

Buildings are supposed to evolve in an undefinable, organic process. Modern cookie-cutter buildings are a symptom of not doing that; rather than an indication that building buildings doesn't require deep thought.

And by extension you can apply the same arguments to most professions: even if we (as software engineers) don't really understand what they do. Neither do they.

I think anonymous (#1) is correct. Most what passes for design in other professions is reuse of existing designs. That is the reason why every office looks the same: it's one solution applied again and again, reuse with only limited configuration.

Configuration is not programming.

Configuration can be made into a process, it can be automated. One can write a program to do configuration. Even if you don't really understand what you need you get a long way with a evolutionary algorithm and a good test set.

Programming is very different. Programming is the creation of original designs. Originality as far as I understand it can not be turned in to a repeatable process.

Your comment "insert miracle here" reminds me of a Scientific American article. Frequently SA receives letters form people claiming to have designed interstellar starships, inevitably there will be an empty box somewhere that say, "put warp drive here."

It reminds me of a speech I heard, somewhere - that McDonalds creates a set process for making a burger. It won't be as good as a burger by a Master Chef because the master schef takes tons of little variables into account - how old the meat is, how fresh the lettuce, etc. MscDonalds makes the process so anyone can make a burger and have it turn out the same. It's not very good, but they are all not very good in the same exact way.

Your post makes sense, but following it through with the above - if we were to create a set process for programming, it could flatten the curve. Poor programmers would become better - but would fantastic programmers become worse (or more likely, fewer in number)?

I'm trying to figure out the point. Is it that, because of lack of process, the disparity amongst programmers is so large? Even if this is the point, it begs many questions. Is the disparity larger than in other fields? What are other possible causes for the disparity?

I don't discount that lack of process may be a major contributing factor toward the disparity between the best programmers, and just average programmers. But in my experience, there are at least 2 other factors, the first being sheer ability. As a basketball coach once said, you can't teach tall.

But the other seems to be motivation. I don't know how many times I immediately see my way through a problem that average programmers struggle with, because I was motivated to read or learn in the past, and so the solution is at my fingertips. The sad fact of the matter, is while excellence at programming pays, mediocrity at programming pays almost equally well, and because of this, our field will always be rife with un-motivated programmers satisfied with simply collecting a paycheck.

Most engineering disciplines create one thing (e.g. a bridge, a building, roadways, etc.) and this one thing is well studied and known. It is repeatable. A process exists to transform the inputs (raw materials) into outputs (structures).

A program is a process by which a computer transforms a set of inputs into a set of outputs.

Get over yourself, this is how all the technical disciplines are. Sure there is a process for a lot of things, but every field has its "non-process" part. The intuition part is crucial. You are saying that we know less about how to do computer science than chemistry, physics, math, or other engineerings? Take a class in serious dynamics and tell me it is less rote process than writing a program. For a large majority of programmers it really is open Java library, duct tape together. I'm not saying computer science is easy, but the truth is all of the disciplines look easy from the outside, to think that computer science is the ONLY one with that magical step is laughable. That magical step is known in other fields as intuition or design. And if you want to look at an institution that has totally removed the intuition step from programming and made it all process successfully, look at NASA. It can be done, it just takes more patience and more money than most companies can justify sinking into a product.

However I think PROGRAMMING in itself is an extensions towards mathematics (formulas, logical thinking even though this is not only mathematics, but it has strong roots in mathematics etc...)

And no, i dont mean statistics. Everyone can learn statistic easily. I mean mathematical THINKING. Where you know so much that you can think in it with ease, while you program.

I also agree about the poster that wrote that programming is about design.

For me, I am using ruby because I feel that the ruby way is (syntax) very clean, while completely staying towards a true object system. Ruby is like a modern merge between perl and smalltalk. Everyone who knows perl, should give ruby a serious try.

There isn't a process for writing a song, or painting a picture either. You can make processes for doing these more creative tasks, but you won't always end up with desirable results.

I wonder if programming is more akin to an art form than a skilled trade. Maybe it utilizes the abstract thought processes and creativity more than it uses memorization and application.

I come from a music background. I have written a lot of music and am just now moving into programming. I have noticed I have a much easier time actually writing programs than a lot of my class mates. Everyone seems to understand the concepts behind the programs, and they know the applications for the concepts because they have memorized and studied them, but when it comes to writing a program they struggle to apply the concepts they have memorized.

I think this disparity can be made up for in some ways. Design Patterns can be learned and applied to fitting situations, just like a Design Pattern can be used to write a song (verse, chorus, verse, chorus, bridge, chorus, end) but this doesn't mean you will have a good song when you get done, because in the end the song is being judged by a humans sense of aesthetics. Good programs are ultimately judged according to human aesthetics as well... whether or not your program solves the problem is just the first metric, but how well it solves that problem, how elegantly, how quickly, and how easy is your program to use.

These aesthetics may be more measurable than the properties that make a good song, which may make a process or design pattern easier to teach to others, but in the end it is all a matter of human aesthetics you are catering to.

Maybe because there are so many surrounding processes we forget about some basics in programming: Take end-to-end responsibility for your code, understand how it will be used, and make sure you are (or someone else will be) able to change it later.

I agree that writing code is a creative task. You need to have a strong focus on the aforementioned basics to avoid getting completely off track (like the worst programmers do).

I can't believe all the people saying all the offices are all the same. I picture my mom saying Ubuntu and Windows are the same because they both have shortcuts on the desktop, drop down menus, off button ...

Excellent point you've made here. I'm trying to come up with a way of describing the process of programming and I'm drawing a blank. The best I can say is that it either happens right away, or first I do some Googling on the parts I'm not sure of, read some things, and then it happens eventually when I've learned enough to synthesize together to solve the problem.

There is actually a "program". Aristotle wrote about it - but it has been a taboo since the church banned his description of the thinkable process of how we create in the middle ages. I have the vital ingredients in my digestion from 2005: Get Rid of Our Final Taboo! - The Human Ability to Create is Thinkable:

Test Driven Development is a process for programming. So is Dijkstra's method of Correctness by Construction. There are similar processes for design at that level of detail in a variety of disciplines.

It's easy to make assumptions about other professions. Your point is well taken that non-programmers just don't get how it really works. I'm with you there. But then you proceed to apply a double standard and assume you know how every other profession works to make your claim that programming is different than everything else.

As an entrepreneur, I've worn a lot of hats. I can tell you that the "absence of a process" phenomenon is a lot more common than you'd think. Marketing, consulting, sales... none of these have learnable, repeatable processes. Sure -- there are frameworks, but they're just that. They spell out what you do but not how you do it.

Most of the posts appear to confirm the 'hand waving theory' of software development: stare into space, become enlightened, commit abstract thoughts to symbols on a screen, cross ones fingers that it is a functional program.

When I do software development, it is something similar to the 'scientific method.

Step zero point one: spend time learning about data structures and program flow

Step zero point two: spend time how a specific language implements tis flavour of data structures and program flows

First step: what is it that needs to be accomplished?

Second step: what are the user actions, what is the data source and format, how does it need to be manipulated, and what is the output?

Third step: by selecting appropriate data structures to match data inputs, calculations, and outputs, program statics can be implemented

Fourth step: Figure out the sequence of steps to get the data in, calculate on it, and then get it out.

Fifth step: test the theory and confirm that it matches the specification provided in step one.

Sixth step: pat your self on the back for successfully writing a program.

Does that satisfactorily define the methodology one uses to program?

Of course, the secret is in the appropriate definition of the question, the successful selection of appropriate data structures, and the proper ordering of program flow statements. But each of those should be able to be defined in a recursively more detailed manner.

Coincidentally, software design is also by far the hardest thing (and somewhat underestimated) to master. Rather than all the processes and algorithm stuff at university, I would really have appreciated design and architecture (patterns, anti-patterns etc.) as a first-class... class.

There seems to be a hidden assumption in your post - that we all have the same mental hardware, but some are lacking in mental software. I am pretty sure that the first part is not true.

As a culture, we have no problem accepting that people are born with different physical potentials. I once heard an exercise physiologist talk about Lance Armstrong. He said the average person could train for years, while Lance sat on the beach and Lance would still beat them.

The way I would explain the 10:1 ratio is that natural mental potential in programming, along with hardwork, motivation and a passion for programming, creates kind of a snowball effect on ability.

I also strongly disagree that this phenomena is limited to just a few disciplines. I think it is the case in most disciplines. What sets programming apart is the observability of the results.

One of the many hats I wear is in marketing. There are so many factors that affect the success of a product, it is difficult to figure out how much a marketer contributed to the process.

All that said, I agree with the others who have mentioned the importance of motivation. At my company, top producers are dis-incented. Top producers do not get paid that much more than bottom producers, yet over time the company expects (needs) the top producers to keep producing at a high rate. This can be really stressful. Every time you do something amazing, that becomes the expectation for you. After a while, there is a strong incentive to stop doing amazing things. Maybe instead of looking for ways to make the programming process more teachable, companies would have more luck creating incentive structures that would encourage higher productivity.

Culture, I think, is the key reason why we don't want to admit that we know how to program. Another common problem is that most programmers make it harder by over-thinking their solutions. It is 'harder' until you learn that it isn't.

I also think that using a process analogy to other professions, you ignore the concept of quality. There may be a structure and process to creating a legal brief or a medical diagnosis, but following that process does not insure a quality legal brief or medical diagnosis. It can mask a poor quality solution. Just as a working application can mask poor coding.

There is another profession that trains by copying ever previous works. Starting with small exercises and working upwards. It's called art school.

And the reason is simple.

Most of the skills that most professionals use are basically analysis skills. They have a bunch of rules for breaking things down and then reacting. Be it visual recognition of road signs, or a set of rules of thumb for negotiating a balance sheet and a cash flow statement.

But all design is synthesis. Take a set of inspirations and a set of constraints and create something new that fits in the hole.

That is why programming is unpredictable. Some people can see the solution, and some can't. And a whole lot of variation on the quality of the solution envisaged by those who can.

I thought this was fairly obvious, but nobody seems to have mentioned it. Imagine you're trying to write software to solve some problem - e.g. adding two numbers. Let's pretend there exists some "process" to write an algorithm to solve said problem. What would it look like? (at a fairly high level)

1.) tell computer to accept two numbers as input.2.) tell computer to add second inputed number to first inputed number (How/what level you do this at isn't really important)3.)tell computer to output the number resulting from this addition.

Something is fishy about this, of course. It's that imbedded in our "process" is exactly the algorithm that we wanted to write. In other words, any "process" one might follow to write an algorithm is going to contain the algorithm to begin with. How you actually go about commanding the computer to do this is a question of syntax. This should look familiar from Theory of Comp.

This reminds me of the moment I decided to leave my job and look for a better opportunity. I was sitting down in a meeting with my boss discussing a particular project that I was about to start working on. Then he asked me to explain to him (a non developer...never was never will be) exactly how I was going to solve the problem at hand. He wanted right then and there a step by step plan of how I planned to solved the problem. That was the moment I realized he didn't get my job and never would.

As to the suggestion that mathematicians are good programmers, I've seen no such correlation. There is too much arcane knowledge and peculiar trivia involved in software development for anyone with an exceptional grasp of logic to have much of an advantage. It is a sad fact that when working on software, patience and general creativity often get you further than sound reason. As an aside, in hiring for our own team we have found those with applied physics backgrounds have the best academic foundation for software engineering.

I disagree, with very much you wrote. At first building houses is a variation of one theme. Sure there are different houses but the principles are "dictated" by sheer needs. For not wasting material you try to centralize things. One central point in many houses is where the heating takes places you hardly find the heating room on one side and all the consumers on the other side.

However another example is farming. You can do follow every rule but may fail, there are things one can not control. If e.g the weather turns out to be bad, you loose a whole year of work.

Or see craftmanship. One can surely automate let's say building a chair, but if you ever have the luck ot find a masterpiece of a cabinetmaker you hopefully get the difference.

Or yet another example, why are Bach, Beethoven and the like are so famous? Why not you if "making" music would be that easy....?

I'm sorry, but your first impression was the right one. "How could anyone be so dumb as to suggest that?"

That does not mean that this is absolutly pointless, but you can send the programmers to as many course about design. If he does not care about his work and "just slaps together code" without appreciating than, he also will underperform.

If you find someone enthusiatic about programming and keen on learning then course might help. But be aware of:http://www.joelonsoftware.com/items/2008/05/01.html

Programmers should know better than anybody that you take a big gamble by saying "my job couldn't be done by a machine", which seems to be what this article boils down to.

Just because you intuitively know that something dropped will fall doesn't mean that isn't a law of gravity that can be taught. That people got by for millennia without it doesn't negate the huge advances that were made once it was discovered.

At the weekend I was discussing with my mother the differences between the Java coding I do in Eclipse today and the punch tape programming she did in the early 1960s, including literally cutting and pasting together sections of programs with scissors and glue. There are so many fundamental principles of modern programming that are inherent in the language and the IDE that we take for granted and which just didn't exist 40 years. I am sure in another 40 years time my son will be amused by the clumsiness and naivety of the way we do things today.

Nietzsche recognised this truth in the 1880s:

"All beings so far have created something beyond themselves; and do you want to be the ebb of this great flood and even go back to the beasts rather than overcome man? What is the ape to man? A laughingstock or a painful embarrassment. And man shall be just that for the overman: a laughingstock or a painful embarrassment. You have made your way from worm to man, and much in you is still worm. Once you were apes, and even now, too, man is more ape than any ape."

It may be safer not to say "It can't be taught" but rather "I couldn't teach it".

I've seen plenty of people comparing to various 'creative' endeavors, but in reality, writing software is fundamentally an act of *writing*.

Writing spans both more and less creative acts, from the mundane grocery list through driving directions (which actually does have a creative element when you are careful to give people the directions that they actually need (turn-based vs. landmark-based, etc.)), all the way up to stories and songs (which also can be subdivided into more and less creative works).

And yet, the process of writing prose is *identical* to writing code (except you don't have a ruthless test for correctness), and the method of learning to write prose is also identical to learning to write code.

Non-coders just assume that what we're doing is equivalent to writing a grocery list (or, at most, driving directions), when really we're usually writing narratives from short stories all the way up to multi-author shared-universe anthologies, because externally it looks the same.

So, why don't non-authors make the same assumption about authors? Many actually do, until they try to write their own Magnum Opus.

Conclusion: to raise the status of programming as a creative endeavor, we need to promote the teaching of simple coding as a basic skill, exactly the way we teach reading, writing, and mathematics.

I'm new to studying software engineers, but I'm not new to studying other professionals. My first reaction to your post is to get angry (again) that someone in software engineering thinks that software engineers are somehow special and different from everyone else - they do a job that doesn't have process. But, getting angry doesn't do much good. So, my second reaction is to think, "Huh, isn't it interesting that we assume other people's activities can be described in process documents?"

Paul, you wrote, "In every case the core of the activity is well understood: it is written down, taught and learned." That's simply not true. Professional activity can't be reduced to process documents, can't be written down, taught, and learned in the simple manner you seem to think characterizes everything from driving a car to running a company. It's not just design work that's complicated.

Please try to be more respectful of others' unique expertise. For more info on how "process" might be a word we use to describe things after the fact, see Lucy Suchman's work on plans (1st and 2nd editions). For more on how expertise develops that has nothing to do with written processes, see Jean Lave and Etienne Wenger's work on legitimate peripheral participation. The midwives, tailors, and canoers they study should fall somewhere between driving and running companies. Not your cup of tea? How about professional sports and less academic writing - becoming an NFL place kicker