I was very happy to take part in the coding dojo of 8 February 2006. The previous time I had attended was the first public session Helsinki in November, and compared to that, the recent dojo went considerably more smoothly:

the cooking stopwatch worked excellently for keeping each person’s turn at ten minutes, with one of the pair rotating every five minutes. (My personal goal for the next dojo is to learn how to set up the watch correctly ;))

the audience kept moderately quiet, and the questions were less suggestive than before — i.e. more questions, less directions

And again, I learned valuable things on how other people mould the code, think about the micro-level design, and write tests.

The word “tests” just above should disturb you, if you think that we are practising Test-Driven Development (TDD) in the dojos.

In the randori-style dojo, as a pair produces code, everybody watches it on the projected screen. Sometimes the audience gives slight signals of appraisal, especially when a pair successfully completes a feature, runs the tests, and the xUnit bar turns green. I wanted to cheer not only for the green but also for the red bars. People found this strange, which bothered me, but regretfully I forgot to bring this up in the dojo retrospective. Now I’ll explain why I like the red bar in TDD.

I’m not sure if Behaviour-Driven Development adds to TDD something more than just the change of the mindset and the vocabulary, but my dojo experience got me thinking that this might be more important than what I had understood. Consider the following (Ruby poker hand evaluator test with Test::Unit):

For the record, for this rSpec version to work, I had to add this method to the Hand class:

def evaluate_to?(hand)
return evaluate == hand
end

While Ruby and BDD might or might not be cool, the real point I want to make is that even without the BDD twist, TDD is about design. So what we should practice in a TDD dojo is how to design by writing executable specifications. I think that this is a fascinating, useful and non-trivial skill that is best being rehearsed when working on small and simple examples, such as the tennis scoring and poker hand evaluator which have been the assignments in the Helsinki dojo sessions so far.

Now we have been talking about getting more real-life kind of problems to the coding dojos, so that the participants could learn how to do TDD or at least programmer testing better in an everyday work environment with application servers, databases, networks and whatnot nuisances. Certainly such a hands-on session would accompany well the excellentbooks on the subject, and help people in adoption of developer testing, but I think that they would be more about the hands-on dependency-breaking or specific technology skills than design.

So although I welcome the idea of exercising in a more realistic setting, I hope that the randoris for doing simple katas will continue as well.

If you dig a ditch slowly, you make progress anyway. But when programming, even if hacking away a dirty solution might seem quick, it might backfire in hindering your progress in the long run. And when this happens, it wrecks the best intentions of agile project management to predict reliably.

Scrum is the best thing that happened to software project management since the invention of sweets in meetings. Scrum frees developers from excessive meetings and paperwork, and helps them to communicate their problems and feelings. And most importantly, the sprint burndown chart provides a good, empirical estimate on whether the project is on schedule or not.

I will now dwell on my last point about Scrum — or indeed any agile project management — needing good programmers. This is by no way a novel nor original idea. For example Pekka Abrahamsson from the VTT of Oulu underlined that agile software development is not for beginners in his excellent “Agile Software Development” presentation in the inaugural Finland Agile Seminar of March 2005. But I must confess that I haven’t deeply accepted this view until recently.

I used to think that agile methods might help just about any project or programming feat, and specifically that even junior developers could benefit from them. And surely they can. But I will now try to explain why I conclude that an agile project needs skilled programmers to succeed.

At the heart of agile project management is the idea of linear, quantifiable progress: the velocity of the project can be measured by the rate of completing the wanted features, and this is used to predict the schedule. This is exactly the function of the sprint burndown chart of Scrum, mentioned above. The developers break each feature down to measurable tasks, estimate the relative sizes of the tasks, and the process tracks the rate of completing the tasks. And this is how you get data on the velocity of completing the features: if making the full text search is a task twice the size of making the login screen, and making the login screen took 17 hours, by all probability making the full text search will take 34 hours. Scrum rocks because it gives this kind of constantly-improving data with minimal effort.

Instead, what we software developers develop is actually a collection of source code listings that do not serve for the end purpose of our work until they are built into machine code that is being executed in a computer. And to get there, the software must compile, run, and when running do useful things.

This is where the non-linearity comes in. When making more features, it is all too easy to break the features that were supposedly completed earlier. It’s very easy to break the source code in such a way that it does not even compile to executable form anymore. But most importantly, it’s too easy to grow the system in a way that increases the cost of changes by time.

At least since Kent Beck’s first 1999 edition of Extreme Programming Explained: Embrace Change the idea that the cost of change does not need to increase by time has been a central agile tenent. And Scrum seems to assume that the cost of change within the project remains constant, because there is not any kind of a coefficient for the velocity that would be a function of time. In my login screen / full text search example, it should not matter in which iteration of the project the full text search would be implemented.

Robert C. Martin (“Uncle Bob”) has written a great analysis of the problem of the increasing cost of change in his book Agile Software Development: Princples, Patterns, and Practices (2002). You can find what seems to be more or less the same text in his article “Design Principles and Design Patterns” (PDF). Here Uncle Bob neatly slices the awkward feeling of “this code is difficult to work with and expensive to change” to four distinct symptoms of rotting design: rigidity, fragility, immobility and viscosity. He then presents the principles of object-oriented design and demonstrates how they contribute to keeping the design of the software such that the cost of change is being controlled.

The more I learn about object-oriented programming, the less I feel I know. Reading Uncle Bob’s book was an experience similar to studying logic at the university: After the first couple of logic courses, I felt that I had mastered all that there was special to logic, being able to work with both syntactic and semantic methods of the first-order predicate logic, and vaguely aware that “some people do weird things with logic such as leaving out the proof by contradiction.” It was only when I understood the alternative and higher-order logic, proof theory and model theory a bit deeper, when I realised that I had just been climbing a little foothill, from the top of which one saw a whole unknown mountain range with a variety of peaks to climb, from Frege’s appalling syntax of predicate calculus via Hintikka’s modalities to Martin-Löf’s constructive type theory.

Making good software design — which, if you believe what Reeves writes, is the same thing as programming well — is difficult. It takes a lot of time, practice, reflection and study to advance in it. And there will be a lot of moments when a programmer overestimates her capabilities and writes something that she sees good but which turns out the be bad design, the debt of which the software incurs to be paid later.

Hence Scrum does not work in a vacuum: if your team does not program well, the progress is not linear as Scrum assumes. The design rots as the codebase grows, this slows down the development of new features and changing old features, and the predictions that Scrum provides start to err on the optimistic side.

I used to think that Uncle Bob’s Agile Software Development had a bad title despite being one of the most valuable programming books I know of. I thought that the subtitle Principles, Patterns, and Practices was a lot more indicative of the content. But with the line of thinking expressed here, I am coming to the conclusion that in the end, good software design is an intrinsic part of deloping software in the agile way, because it is necessary for controlling the cost of change over time.

And we made changes accordingly to the FooController class, for the case that was needed in foo.

Then we started getting NullPointerExceptions from fooBar and fooBaz when they were executing the code of FooController that required baz, even though the views of fooBar and fooBaz did not need it.

So, even though Spring beans are by default singletons in the Spring sense, the Spring singleton sense is rather like Uncle Bob Martin’s Just Create One. Different beans having “singleton” instances of the same class will have their own instances, so providing the dependencies to the foo instance does not help the other instances.

Noticing this kind of errors earlier in the development is the reason why I prefer initialising the crucial dependencies of an object in the constructor, preferably making them final members. This way you cannot create an instance in a crippled state in the first place. Speaking fancy, this would be the “type 3” or constructor-based dependency injection. And for more complex dependencies I like context IoC as well.

Anyway, for a little simple problem like ours, we wanted to have a quick, simple solution. Here is the list of solutions we came up with. If you have more ideas I would be thrilled to hear them!

Separate (sub)classes for each controller. Some day they will surely grow separate, but at this moment I felt awkward about littering the class space because of what is basically a configuration or dependency wiring issue.

Making FooController a traditional singleton by putting factory-method="getInstance" to always return the same, statically held instance of the class and making the default constructor private. But this would have been both verbose in the Spring configuration XML and again littered our java code because of a wiring issue.

Adding the dependecy wiring to each controller bean entry in the Spring XML. But this would have caused duplication in the XML.

For now, we settled for the last solution that seems OK to me. Sometimes I have thought that the parent-child-hierarchy in the Spring configuration XML might make it too complicated, but in this case it feels justified and the configuration keeps looking fairly simple.

I was a fan of the stories of Thor Heyerdahl — I read Kon-Tiki, Amon-Ra and also a book on the mystery of the huge statues on Easter Island. The scientific-adventurous style of the books is great. But what I will write about now is a little curiosity that stuck to my mind in the Ra story.

Ra was the Papyrus reed boat that served in one of Heyerdahl’s famous feats of experimental archaeology. He wanted to test the hypothesis that the ancient people would have had vessels capable of crossing the oceans. Earlier, with the balsa raft Kon-Tiki he could show that the ancient South Americans could have travelled all the way to Easter Island. And with Ra he intended to show that the old Egyptian reed boat design was capable of crossing the Atlantic ocean.

Heyerdahl had the reed boat built following the designs from the old Egyptian paintings. Many of the paintings showed a rope between the bottom and the upward-tilted back tip of the boat. Heyerdahl thought that the function of the rope was to force the back end of the boat curve upwards, but when he observed that the boat stayed in good shape even without the rope, he decided to leave it out. This proved to be a bad idea:

A decision to cut what proved to be a crucial line between the prow and the deck caused the stern to sink and the ship to increasingly list to one side. After almost two months afloat, and with a hurricane approaching, the Ra was abandoned at sea frustratingly short close to its goal, still floating but nearly uninhabitable.

Heyerdahl learned valuable lessons, built a new reed boat Ra 2, and crossed the Atlantic with it. In the realm of commercial software production, this might be equivalent for example with a quick and easy bug fix, or with a major financial failure, depending on variouscircumstances.

Obviously you would not be reading even this far if you did not find some interest in fabricated parallels between software and the real world, such as this: the mistake of Heyerdahl is like leaving the programmer testing out of XP. More generally, even good advice can turn against you, if you make a misinterpretation or false assumption, and end up not doing quite what the advisor was thinking about.

The extreme programming example goes to the heart of what I want to convey. If you take such a broad and complex holity as the extreme programming, and do not realise how the separate parts need the support of each other, you might inadvertently cut off a crucial line. This might well be provoked by an illusion of resource scarcity or other types of laziness, for which we have some great expressions in the Finnish language: rusinat pullasta (“only the raisins from the sweet buns“) and yli siitä, mistä aita on matalin (“crossing the fence from where it’s lowest”). We Finns pride ourselves on eating the boring dry doughy (domestic) part as well as the sweet (imported) raisins, and on crossing the fences at their highest parts to not look like slackers. The kind of slackery of not jumping that high when it’s not exactly necessary for getting on the other side is nowadays fashionable as effectiveness, and I think that it is exactly the pursuit of this kind of effectiveness in which you might go astray.

Already in the first edition of the “white” extreme programming book, Kent Beck underlined the fact that the different legs of XP support each other and that the process would fall down like a house cards if you left one wall out. (Without still having gotten around to read the second edition, I should mention that it’s most probably well worth a read.) Obviously he did not describe his process like a house of cards, but that’s the feeling that stuck in me. Later on, I think that Beck pretty much confessed that of the XP practices, at least test-driven development could stand alone, so even if it might not save your project, you might be able to do good work practicing TDD even in a project that does not embrace other XP practices. Though if refactoring would be limited politically, this might make it very diccicult to do TDD.

Of the other XP practices, I think that perhaps pair programming might fly alone as well. Without proper first-hand experience, my guess is not educated. But something I have been through is what happens when you take the XP path of leaving out the big design up front or indeed almost any design up front (which is already getting too extreme to be proper XP), but fail to implement automatic testing, merciless refactoring and pair programming. If this doesn’t result in a mess for you, you better play a round of lottery. It is exactly this kind of faulty shortcuts that give “XP” the bad reputation as glorified hacking.

Another, more subtle example of the intersupport of XP practices, is the crucial but easily overlooked third step of test-driven development: red – green – refactor. If you leave in the repository the first kludge that seems to work, you’re not actually doing TDD but causing trouble:

Once the tests pass the next step is to start over (you may first need to refactor any duplication out of your design as needed, turning TFD into TDD).

To keep your TDD boat afloat, you need to refactor both the production and test code continuously. It’s not enough to just write a test always before writing code. Another necessity is that all tests are run often and showing green, as maintaining an exhaustive suite of tests is a part of TDD.

Another example still from the XP / agile realm, but from a bit higher point of view, would be overconfidence on developer tests as a measure of progress. For me so far, customer-defined automatic tests or indeed any predefined customer tests have been but a dream. And to measure whether the software is close to complete, it is exactly the customer / user tests that matter. Even if you have good code coverage by programmer tests and you think you have implemented a good part of the required features, you must fight to get the feedback from the customer, or whoever in your environment would be closest to the real user of the system.

What I still saw, even with TDD, are misunderstandings between the project’s customers (also known as business experts or product owners) and the software developers. Even if the deployed code was almost bug-free, it didn’t do everything the customer had expected. We can enlarge on the concept of TDD and reduce the risk of delivering the ‘wrong’ code with what I call customer test-driven development.

To make all these long stories one short one, I claim that you must not adopt a process or a method, and then leave out a crucial part, because then you might fail horribly. Am I then saying that you should adopt any good-looking process blindly and without criticism?

If you got this far and think so, I hope that you will come up with a general counterexample in the next few seconds 🙂

Well, of course I don’t advocate following processes blindly, because cargo-cult programming can be just as bad a problem as taking the faulty shortcuts. In fact I think that the problems are related — omitting crucial parts of a method can be seen as doing cargo-cult on the remaining parts, without fully understanding their dependencies and interactions.

So in the end my point here is not really about being lazy in adapting practices, but being lazy in thinking about your work. The steps on the journey of adapting a method or a process effectively can be seen as the stages in reading texts on a subject with which one becomes more familiar, and can thus move from restatement through description and interpretation to critical reading. To master a skill, you must have the passion, perseverance and ability to learn, so that you will build a couple more reed boats before you get one to cross an ocean with.

We have gradually introduced a little Spring in the system I’m working on, and the results have been good. It’s getting easier to write testable code that can be read and modified efficiently, which is saving our customer money. (My account manager told me to end every paragraph of whatever I say or write in “saving money” or something such.)

However, the silver bullet of Spring does not come without feeding problems. A while back I was making a simple Spring singleton service that stored state that was periodically updated. The problem is that the state did not refresh. In despair, I fell to the despised resort of debug logging. This showed me that the internal state of the service was indeed refreshed, but it did not seem so when calling the service from outside. What?

At some point it occurred to me that maybe my singleton was not really a singleton, and that maybe I was refreshing something else than the object that the rest of the software was looking. And indeed a simple test showed that this was the case.

The Spring ApplicationContext was being refreshed in the bootup process, but unfortunately this was after non-Spring legacy code had acquired a reference to the original “singleton” service. Obviously the context refresh could not do anything to the references from outside the Spring application context, so we ended up with two copies.

Fortunately this one was easy to fix. I also added to the application context a simple bean that will throw an exception if it is initialized twice. This causes context reloads to fail with the exception, which might be a good thing as long as we have non-Spring code that will hold on to stale instances of Spring beans on refresh.

I am starting to write a weblog, or more accurately writing the first post that will hopefully have something that follows it, because writing about what I do and think is something that helps me a lot in reflecting on it in a productive way. This must happen a lot with us literal types that need to struggle to say a few a words, let alone intelligible ones, but will joyfully whack away a couple of several-hundred word emails on whatever triviality that makes us twinkle.

Writing is also the obvious form for a programmer to express what she does and thinks. A lot of what is traditionally seen as programming is effectively doing thinking writing, especially nowadays with the exploratory mock-powered test-driven development that allows one to refine her ideas on the module at hand by writing code. In the end of the day, writing code is a big part of what they expect from us. And because it’s important to think about what you write, it’s great when you can write in an exploratory way, thinking by writing. Perhaps word processors and digital cameras have directed natural-language writing and photography to similar directions as the modern software development tools have directed programming:

In reality, the act of writing the source code has never been the only necessary part of producing good software. Thefocus on people does it for me big time in the agile school of thought, albeit accelerating the feedback cycle on all different levels of actions is at least as important. I want to excel as a programmer, an emphasis that some much more highly esteemed software professionals underline, steering away from management and office applications towards coding and development tools. But as I believe that face-to-face communication is the best way to educate the programmers on requirements, I strive to keep close to the stakeholders even as a mere “coder”.

I intend to use this blog as a vessel for reflection on programming as a real-world activity.