Wed, 15 Jul 2009

Musings on Rakudo's spectest chart

In an attempt to track progress in a measurable way, the Rakudo people count the number of passing tests
each night (ok, they don't do that manually :-) and write that into a
CSV file.

The next step wasn't far: generate a chart from it, put it on a website, and tell everybody about it. Here's
a snapshot:

Old news so far, as an eager Perl 6 follower you surely know that already,
nothing new here. But the other day somebody asked me why the green (PASS) and
the grey (number of spectests available) seem to be parallel.

My first reaction was "no, they aren't parallel" because the big jumps
don't happen at the same time, but still there are large areas where they do
seem to grow roughly at the same rate. So what's up?

As far as I can tell there are two mechanism at work here. The first one
accounts for much of the past behaviour: tests were moved from the old part of
the pugs test suite to the official test suite (where they are counted towards
the grey line), and at the same time people also tried to get it to run under
Rakudo - mostly successfully, as the green line shows.

Now most test files are actually moved to the official Perl 6 test suite,
there are less than 200 tests left to move (as opposed to 17k already
moved), and this mechanism is drained.

The second mechanism is that as soon as Rakudo implements a cool new
feature, we realize that we need more tests for it, and people write more
tests for it. (It's rather hard to write tests for language feature that
doesn't run yet. "test first" is a nice idea, but it only works to some extent
in reality). Also people explore the new features, find bugs, and bug tickets are
only closed if there are tests for them, so even more spec tests that coincide with
passing Rakudo tests.