On Thu, Jan 22, 2009 at 05:57:08PM +0100, Peter Bowers wrote:
> Pm wrote:
> > I'd welcome a way for us to have automated tests for PmWiki,
> > and I think we can support it on pmwiki.org. However, I
> > warn that it's not as trivial as it sounds at first, and
> > someone other than me will need to implement a prototype or
> > proof-of-concept.
>> Perhaps if you share your approach it might get some creative juices
> flowing...? In that spirit I'll offer a thought...
My specific thoughts are below, but the overall approach I was
taking was to create wiki pages containing markup and the
desired HTML output, and having some specialized actions
that would run the tests and report the results. In the earliest
versions I was doing each test as its own wiki page, but
that quickly became unwieldy. So then I switched it to allow
multiple tests on a page, and grouping tests together by topic.
That worked better, but keeping tests up-to-date was a big
pain, and it was also a pain to generate summary information.
Generating summary information might be a lot less painful
now that we have pagelists (which we didn't have at the
beginning of the 2.0 series).
Perhaps that's a sufficient enough test for our purposes
("did _anything_ change?"), but I think I was looking for a bit
better diagnostic than that -- something that could describe in
better detail what had actually changed.
> Of course the difficulty is not in coding the test script nor in running the
> test script, but in choosing the appropriate test data and checking it by
> hand and then saving off the known-good-copies (although this saving off can
> be automated as well).
As I mention above, I think that hand-checking in this scheme is
going to be unrealistic, because small changes to unrelated features
would require updating every page in the test database.
> ===my very off-the-cuff tho'ts===
> I was thinking through using wget, saving off known-good-html files, and
> then looping through with various configurations on various servers and etc.
> Something like this [caveat - all theoretical at this point]:
[ below excerpted for simplicity ]
> for htmlfile in $domain/html.d/*
> do
> curhtml=$(basename $htmlfile)
> wget http://${curdomain}/${curhtml} #what options needed?
> if diff -q ${htmlfile} ${curhtml} #don't remember how wget names
> [...]
> done
>> Obviously some more error checking has to be thrown in around wget and etc,
> but that should give the basic idea...
For purposes of discussion, let's call the files in html.d/ our
"testing standard" -- i.e., those are what we expect valid tests
to produce.
The biggest problem with this approach is that "diff" is much
too coarsely grained to help us locate problems. For example, if
we simply change a skin feature, HTML header, or even just the amount
of whitespace that occurs in certain elements, then the above would
report that every test has failed because it's no longer _exactly_
the same as our standards in html.d/ . In other words, the "failures"
it reports aren't really failures in PmWiki; each of the pages are
still semantically correct -- they're still rendered properly in
a browser -- but the diff command falsely reports them as having
failed.
In order to get around these failures, we'd then have to update
all of the files in our "test standard" (in html.d/) with new
copies of what we expect. That's too onerous to do manually,
and if we do it automatically then we aren't really testing
for "correctness", we're just testing if the output is changing
at all from one version to the next.
Perhaps having a system that tells us if _anything_ changes is
useful, but I was aiming for something a bit more fine-tuned than
that.
> Of course the difficulty is not in coding the test script nor in running the
> test script, but in choosing the appropriate test data and checking it by
> hand and then saving off the known-good-copies (although this saving off can
> be automated as well).
As I mention above, "checking [the test standard] by hand" is likely to
be impractical with this approach, because it would have to be re-checked
for many otherwise insignificant changes to the core code.
Pm