Preparing your tests

Hmm, I just realized, that it's been a while since my last posting here. So, hi there! And for a difference, this time it isn't about java quality :-)

Last month I was pretty much busy with our testbase build automation. It's not a secret though, that our server VM testbase has a lot of native test code. As David hadmentioned recently
we have quite a handful of platforms to take care of and it's getting even more troublesome when you need to carefully lay out a lot of native bits.

So, here some insider's information, and please don't tell any one where you got it: we need to build the native pieces for as much as eight platforms (I might've missed a one or two), including Solaris/Linux/Windows on AMD64, of course. Most of native build's pieces are getting built on a separate machines, e.g. you can't build Solaris for Sparc and Solaris for x86 on the same box, can ya? Well, technically speaking, you can do cross compilation in most of cases, but we aren't that limited with hardware resources. Any way, doing this kind of drill manually might be a boring, not cost-effective effort and one can definitely spend all this time for something moreinteresting

So, we decided to automate out build process. The whole thing was divided into two pretty isolated pieces: writing Ant managed build mechanism and setting up a tiny run-n-watch framework to control as many as 6-8 remote processes, spawned in a certain order. The Ant part has been taken care off by our genius engineers (sic! I mean it) in Saint-Petersburg office and I handled relatively simple execution part.

We do use two different distributed execution systems in our daily business: DTF and Grid engine. Latter takes care about Unix's and former handles Windows, because it's pure Java and doesn't care on what hardware it's running. As the result, I had to create two different sets of starters and watch-dogs to manage our build processes in a slightly different manner.

And I was kind of surprised, that I can write distributed watch dogs, using nothing but Bourn shell (yes, the matter of fact, you can actually program on it and create clear and maintainable code. Wow!). Of course, there's no inter-processes communication or multi-threading. I didn't use anything else more fancy than file-locking (I know, I know â€“ I hate this myself). But it was sort of fun to do this. Overall, the watch dogs' system is quite simple:

- there's a single driver script, which knows everything about build logic and sequences; takes care about any clinches or time out issues

- it sends a set of smaller platform specific scripts to Grid'ed and DTF'ed machines

- sets a number of watch-dog monitors to trace build progress on every platform

- platform specific build drivers do an actually Ant build and are sending notification to the central Tinderbox-like monitor. This helps to ease visualized human control over the process.

- top level script is rolling in a big loop until for all process to finish

- upon the build is successfully completed, the main driver does all necessary adjustments for the fresh bits and run baseline testing against this new testbase. VoilÃ !

That was my pretty much first real experience of using Grid and it unveils some pit falls, perhaps (or it may be me knowing just too less about this system). For instance, you can't rely on relative paths in your application, because Grid copies a submitted shell script somewhere and executes it from there. Eventually, what was `dirname $0`/../logs/mylog when you were developing your script, will lead to nowhere.

Two other things: setting an 32- and 64-bits environment on Windows platform from a shell script. It took me a while to carefully write a shell equivalent of SetEnv.Cmd batch file, which a Visual Studio supplies.

And I had to find a way around the fact that Linux's automounter sometimes has troubles mounting NFS' auto-maps.

However, on a pleasant note: I had spent a good chunk of my professional time between 2001 and 2004 developing DTF and now it's paying back :-) by nicely doing some leg work for me.

So, yesterday I was able to build new Server VM testbase out of the box without much of my involvement (still had to get done a couple of quick fixes, simply because I was lazy enough no to do any dry-runs beforehand). Hopefully, next time it will cost me as much as a single keystroke to start that main driver script.

What will be a conclusion, you might ask? Hmm, I don't know â€“ I just liked the stuff and the way of getting complex things done with a set simple techniques and tricks :-)