The Trials of Testing

In October 2006, we had 1337 tests, and they took thirty-seven minutes to run. Currently we have over 2500 tests and it takes almost an hour and half to run the full test-suite.

Part of the increase in time is due to changes required to make the test suite more reliable [1], the rest is due simply to the number of functional tests we now have. Each functional test starts and stops the application and drives the GUI to test features (user stories) 'from the outside'.

Currently we are trialling just running the unit tests before checking in. Occasionally this causes build-failures when a functional test picks up on a problem that the unit tests miss [2], in which case the highest priority is to fix the build.

This is an unacceptably long time for our test suite to take, and is impacting on our ability to be agile.

One answer we hypothesise about, is a build farm of ten computers and a distributed build-server. You throw a 'build-tree' at it, and it chops the build up into chunks of tests and runs each chunk on one of the machines [3]. That would cut our build time down. As the build time increased we could just add machines.

Really we would want two build farms. We would like our continuous integration build runs to be faster as well, so that we can get quicker feedback about build failures. So having twenty spare machines would be nice.

Unfortunately we can't afford this yet, and there is an underlying problem that is a major cause of the slowness. We have a lot of unit-tests that aren't really unit tests.

We don't test GUI layout. We do test that major components are present, and sometimes the structure of the layout (we actually have a test for one of our layouts that tests that none of the components are too close to each other). Layout is an aesthetic decision though, and not (yet) suitable for automated testing.

We do need to test that events and components are wired correctly though. For example selecting a radio button can cause other controls to be disabled, or pressing a button should launch the correct callback. Testing with very heavy mocking makes for long (in terms of code), ugly tests, that are very tightly coupled to the structure of the code. Testing with a live event loop and GUI thread makes for easier to read tests, but they run very slowly in comparison.

Currently, our build takes 4 minutes, and our acceptance-tests take almost an hour. Therefore, several builds are often produced during a single run of acceptance-tests. In this case, the utility skips ahead to the latest build, leaving some intermediate builds untested. This is preferable to a situation where the acceptance-tests cannot catch up to the builds.

Our build (without tests) takes a similar amount of time.

This is an interesting challenge. I'm not interested in a solution that sounds like 'test less'. The test framework has proved an invaluable tool. As well as catching bugs, the test suite becomes essential when refactoring or adding features. As you break things in the process of changing, the test suite lets you know when everything is working again. Perhaps more importantly, TDD is not just about providing good test coverage, it is a great way to develop and produces quality code.

We do need to look at how we write unit tests, particularly ones which exercise GUI components.

Ruggedized Devices and the Top Ten New Smartphone Technologies

A close look at the top ten up and coming technologies for smart-phones and mobile devices. From flexible displays, to 3D accelerators, to fuel cells; there are interesting changes afoot in the world of communications.

If you're anything like me, then any new and cool gadget comes with a risk that it will soon die from being dropped, squashed or otherwise destroyed.

This article is a look at some of the cutting edge rugged mobile devices, with a review of their features and the pros and cons. There is some impressive looking gear shown here.

These articles are both written by Rene Tse. I particularly like the article on rugged mobile devices. I'd love to get a new PDA [1], and I even kept my last one for two years without breaking it, but having one that can cope with rain and klutz-induced-disaster would be a big bonus.

Embedding IronPython... in IronPython

We recently decided to take Resolver multithreaded in order to improve user responsiveness when processing complex datasets [1]. We're using a fairly standard model: the GUI runs on one thread and calculations are done on another. The results from processing are synced with the GUI using a queue and 'shared nothing' [2].

This works better on IronPython than it does on CPython, because there is no GIL.

Our next user story was to add an 'output window' to the user interface. This allows us (and users), to put print statements in the user code that gets executed when processing datasets (this is part of the genius of Resolver). The prints go to the output window for debugging.

Diverting standard out like this is easy in Python, right ? Just replace sys.stdout with a custom object. Except what if the user has multiple datasets open ? Only one of the windows will be active (the GUI uses a single thread), but it is possible that there could be simultaneous calculations going on. If we have multiple windows and our standard output replacement receives data, how do we know which window to send it to ?

In CPython we could use a hack like sys._getframe to inspect the calling frame, but for a while we thought this might be a show-stopper on the user-story as we had envisaged it.

Except.

The heart of IronPython is the PythonEngine. You can set standard output on an instance of PythonEngine to any stream from the outside. This can be any class that inherits from System.IO.Stream and implements certain methods. That means that we could capture each stream individually.

At first we thought we might have to do some of this from C#, which would be a hassle, but not really a problem. When we spiked it we discovered that we could instantiate and configure new IronPython engines from inside IronPython. That's something you don't do a lot of from CPython!

This has a great side-effect, the code running in the new engine is completely isolated from the engine that launches it. It is so simple that you can even play around with this from the interactive interpreter.

This might be particularly useful if you want to embed IronPython into a .NET application, you can first prototype and experiment from simple IronPython scripts. Have a look at Embedding IronPython... in IronPython for some simple examples of what I mean.

Betting on Python 3000

Since I work for a company producing a 'numerical analysis' tool aimed at solving problems faced by the financial services industry (hedge funds and the like), it is only fitting that I should contemplate 'gambling' on the stock exchange.

Unfortunately I'm not rich enough to play with real markets, luckily there is a fun alternative:

I've also just started to use Grabit, a windows USENET binaries group browser. The client is free, but access to the database of content costs around $2 a month. So far the experience has been good, it seems like an excellent way to browse the binary groups. This may even replace torrents for me.

Resolver is Hiring

But not (yet) developers. Resolver the application is now safely in the hands of our first customer, and we're ready for more. One of the founders of Resolver is a born salesman (and has arranged demos for several potential customers), but he isn't available. We're now looking for a salesman, so if you know anyone available for sales work for an interesting tech product with an even more interesting young company, we have a vacancy.

Resolver Systems are an up-and-coming software development house, with a
product already live in London hedge funds. We are a young company,
and are looking for someone with proven sales and marketing experience
(or at least enormous enthusiasm), who will be helping us keep our clients
and potential clients happy and interested, working with the development
team to give dazzling demonstrations, generating new sales leads, and
creating compelling advertising. A strong interest in selling a very
technical product is essential; experience in similar roles would be an
advantage.

Salary, options package and commission dependent on experience.

Email your CV to jobs@resolversystems.com if you are interested. We still haven't closed the applications for interns either, so if you are a student interested in paid work as a full member of a development team this summer, there is still time.

Towards the end of the summer we may have a new vacancy for a developer, so if you're a Python hacker in London, keep reading.

Hopefully soon I will be able to tell you all about Resolver, and possibly even give you an excuse to do some hacking with IronPython...