Just went to the cinema, and there it was again: Movie hero drives a fast sports car, and shifts gears like mad -- though he has an automatic transmission on board. What the hell does he do? Comfort, sport, comfort, eco, sport or what? I mean, cars like an Audi A8 W12 come only with automatic transmission, for good reason, and then the movie makers still believe it's much more dynamic or what if the driver switches between the automatic transmission modes while evading other cars at 200 km/h. Plus, if the gear changes, you should hear that the engine sound changes as well - that's also kinda hard to understand for them.

Second part is about car performance, if the police is driving a ordinary car with let's say 200 PS, and the movie hero has like 600+ PS, there is nothing to chase for the police when he can accelerate freely. Accelerating with 600 PS is more like teleporting away. In Ronin, it was depicted kinda properly, that's why I like that movie, but in the last bond for example, he could barely escape an Alfa Romeo with an Aston Martin. Come on, that's 3x the horsepower, a much better chassis, better breakes and tires, there is no way the Alfa can catch up with the Aston if both accelerate with full power.

I tend to visit quite a few news sites and forums each day, so I stay up to date with the latest developments. However, I noticed that I'm spending increasingly more time just browsing through news sites, which is bad for the productivity as one gets easily distracted. I've been using news feeds in Firefox and Thunderbird so far, which require more or less manual update. This is even true for Thunderbird, which does check the feeds, but you still need to go into each feed and mark the stuff as read, so the "unread" counter remains useful. Otherwise, you end up with 20 feeds which contain 2000 unread posts, which is hardly useful.

Well, these days are over for me, as I've started using a proper feed aggregator (in my case, FeedReader, but I'm still looking for a .NET or Java alternative with full source code). It has pretty decent filtering, which allows me to skip the StackOverflow questions in my "today's news" feed, and the "mark as read" functionality is right: In my aggregated feed, I read the headlines and interesting topics once in an hour, and then I mark them all as read, which just works "“ each post in each feed is marked as read, and I'm back to a unread-count of zero (or rather, no news items, as I filter out already read topics). Overall, it saves me quite some time per day, and I guess I'll be checking the news less often in the future, as I can be pretty sure that I don't miss something.

Before you get the wrong impression, I'm also writing code besides writing test code, but it's still very much work in progress, so I don't want to talk too much about it yet. So back to testing, which I'm practicing much more extensively now than ever before. What follows are some notes, best-practices and random stuff related to testing.

Unit tests

These are the test workhorses, which make sure your refactoring is not introducing new bugs, that your interfaces are sane, and that the code does actually work. For every project now, I usually write either the tests along with the code, or immediately afterwards. I'm not so much a fan of the TDD approach, where you are supposed to write the test code first: You can easily wind up modeling the interface according to your assumptions about the implementation, which may or may not be valid. I much rather prefer writing the interface first, then the implementation, and then test it. That way, I readily see whether the interface is easy to implement, and while writing the tests, I see how difficult it is to use. Remember to refactor the stuff after a week or so, because one week later, you'll see things in a different light again.

Another thing that you should absolutely avoid is doing sort-of-functional testing in your unit tests. If you have binary data you need for your test, put it right into the executable, and compile it along. It's no use if you get an error while parsing a file on disk, because it does not tell you whether the function you are testing is wrong, or whether it is a random IO problem. Actually, a single function shouldn't both parse and access the disk anyway. If done right, you should be able to run all tests as a post-build step, and they should take something in the order of 1 sec to finish (try to be so fast with IO access).

A side note: If you discover a bug in your code, no matter how tiny, write a test to verify that you actually fixed it. Sometimes, trying to reproduce the bug highlights another bug somewhere else, and you won't find these otherwise.

Functional/System tests

This is also important, and yet often projects don't have an automated setup for this. After having all your unit tests in place, you have to continue and test things like the IO layer, reproduce user bugs and similar stuff. For this, you really need some testing framework which sets up a clean initial state each time you run it, and executes your application, observing its behaviour. Currently, I'm using a bunch of .NET helper applications and a C++ test runner for this, which works reasonably fine. Just make sure that in case of a crash the tests continue and no "Assertion failure" box pops up.

In my case, I pack the test cases into a single DLL, and run them from a test runner, so I can reduce the number of projects to build. Alternativly, you can of course have one executable per test, but this can quickly kill your project build times.

Remember to document this system, so other developer can easily get started with it. I prefer plain text files for documentation, as you can be sure that everyone can read them, but you might also want to use something like Sphinx, which works on more or less plain text. Just make sure that it does not require leaving your favorite IDE to read!

Final notes

So far, I'm getting on pretty well with this. Over the last weeks, I've implemented a new library for my project, which is covered by 50+ unit tests, and I'm pretty happy with the resulting code. Moreover, I could safely refactor some functionality today, being confident that I don't break anything. This is really worth a lot.

Just testing on one platform is not enough. Over the last two days, I've been porting a fairly large project over to Linux, and today I fixed the last problems. So far, all 400 unit tests that work on x86 and x64 Windows work also on x64 Linux, which gives me 6 (!) different versions of my project to test (OS x Architecture x Debug/Release). During the porting, I caught a few non-standard C++ constructs, which could create problems (some ambiguous stuff), but which were trivial to fix.

Much more important was however a misinitialisation, in which I forgot to init a member variable. The class has more than ten members, which are all initialized in an initializer list. Well, all except two, and those two worked fine in every configuration except for Linux/x64/Release. I was pretty puzzled at first, suspecting it might be some compiler bug, as the code that was executed was rather complex (lots of template metaprogramming). Some digging around didn't bring up a good solution, so I thought I'd give valgrind a try. Valgrind is an excellent tool for capturing memory leaks and unitialized variables, and it immediately gave the right hint. Half an hour later, the bug was fixed and everything works fine again now (remember, that's 6x400 = 2400 tests running!).

The lesson to learn here is: Try actively to port as much of your application as you can. Even if everything works fine on your current development platform, it can very well contain hidden bugs, which may pop up later on when the compiler changes on your platform, or if the memory block you place the stuff gets different, or when you ship it to your customer. Moreover, porting highlights other problems as well:

lower/upper-case names: Linux is case-sensitive, and captures mistakes in misspelled header names. Better check often than having to update hundreds of headers when you eventually have to port to a case-sensitive platform. Or if you try to access a case-sensitive web-server.

build dependencies: You see immediately on which libraries your application depends, and you are forced to think twice about using precompiled ones (for example, the distro libraries) or compiling and linking yourself. Also spots problems like "I'm expecting version 1.2 of library X, but my Linux comes with 1.3, which breaks Y, and I never noticed cause Windows didn't come with it at all".

Path-handling problems: A funny one, but some of my build tools were expecting "app.exe", while on Linux, it's of course just "app", and hence didn't work. Same goes for '\' instead of '/'. Takes rather long to fix if not done right from the beginning.

Another interesting possibility that I have now is to run a few of the tools in the project on our Linux server, which is a pretty beefy machine. Previously, it would have required a developer to keep his machine running for hours (or maybe even days), but now we can use the server for this long running tasks. This, and the fact that I could iron out lots of smaller problems, was surely worth two days of porting. I'm also hoping I'll be able to profile the tools on Linux using callgrind, which saves quite some money for VTune and friends.

The title line says it all. Invest time in testing, the earlier, the better. Recently, I've been (re-)implementing some fairly simple class here, with a few unit tests. I tend to test at least the basic functionality with unit tests (i.e. testing good code paths), so I never commit really broken code. I was pretty sure that I did a good job on the class, after all, it was much simpler than the previous one.

Today, I started adding tests for the various corner cases and error reports. Most of my error checking is in the interfaces, so I can easily write a mockup implementation, test it and be sure all derived classes have exactly the same error checking. This one didn't reveal anything new, except for one case, in which the class was not setting a state properly. No big issue, as this was just an error state, and the next access would fail anyway, but nevertheless, I wanted to get rid of it, and it turned out that I could simplify the code a bit. Not much, 5 LoC and a member variable less -- but if you do this for many classes, the savings accumulate.

5 LoC here, 10 there, repeated over 20 classes, and at the end you might end up with a 10 KiB smaller binary. Remember, the fastest and most correct code is the one that isn't there, so I consider this a real win. It's quite interestingly to see how clean you can get stuff. After a few iterations, my classes are much better designed, more robust, require less code to use and have less code to maintain themselves. This sounds like the initial implementation is usually bad; but I tend to disagree -- it's only after having done it that you often see how to do it better, and at the beginning, there is no test net to safely try stuff.

However, it requires really a two-part approach:

Early unit test start. If you add basic unit tests too late, refactoring becomes too expensive. The tests should cover the working cases.

A code refactoring step, which should be done either after a break from this particular code or by another person. In this step, tests should be created to test error and corner cases.

I tend to write code & unit tests concurrently, and after the class is implemented, I document the most important functions. A week later or so, I go back and add unit tests for failure cases, trying to break the class. During this time, I almost always find parts to refactor, either because I see that I repeat code while writing the unit tests, or because I hit bugs. Funnily enough, it really takes a break before I get a feeling for where a class or function may break. This is probably something which can be done better in pair programming, but unfortunately, I didn't have a chance to try that yet.