Re: testing framework and package.el

From:

Christian Ohler

Subject:

Re: testing framework and package.el

Date:

Sun, 03 Oct 2010 21:51:18 +1100

On 2/10/10 13:53, Sebastian Rose wrote:

1.) Make testing a matter of one key-stroke
To ease testing, I've written two little commands, as you can see on
http://github.com/SebastianRose/org-test
`org-test-test-current-defun' and
`org-test-test-buffer-file'
The idea is to search the directory structure from the current source
file upwards for a directory named "tests/" if it exists. Else ask the
user. Similar to what `add-change-log-entry' does.
Below that directory, a tree like the source tree exists:
project
+-- lisp/
| +-- a.el
| `-- b/
| +-- b.el
|
`-- tests/
+-- a.el/
| +-- tests.el
| `-- a-defun.el
`-- b/
+-- b.el/
+-- tests.el
`-- b-defun.el
If this setup exists, when editing defun-x in lisp/a.el,
`M-x org-test-test-current-defun' will load tests/a.el/defun-x.el
(fallback: tests.el there) and execute all tests with selector
"^a-defun".
`M-x org-test-test-buffer-file' in that same source file will load all
*.el files in tests/a.el/ and execute all ERT tests for selector "^a".
This would a) recommend a certain structure for testing (what ever the
structure will be) and b) make testing a key-stroke.

This structure looks like it is optimized for many test files for each
file of non-test code. Right now, Emacs has 1735 .el files, and 35
files total in the tests/ directory. This ratio is not likely to invert
any time soon. The structure that you describe could work, but isn't it
overkill?

2.) Create temp buffers for output
One thing I miss is a way to create output buffers automagically (I
didn't look deeper into the sources yet).
I know Lennart Borgman has written something for nXhtml. It's called
ert2.el and part of the nXhtml distribution (but I did not comprehend
what's going on there yet - my elisp is badbadbad :). Maybe Lennart
will help here or even care for that addition?

Lennart's code is already integrated in ERT, obsoleting ert2. I
remember that I wasn't quite happy with his design, but didn't have time
to fix it, and since it solved an important problem, I included it as an
experimental addition in ert-exp.el.

3.) Compare output to control files
ediff the contents of those output buffers with control files. We need
this feature to test org's exports. No one will find the time to sit
there and search hundreds of output files for errors "by hand" (once
tests exist).

What do you mean by "sitting there and searching hundreds of output
files for errors"? Do you have an example of tests where ERT's error
reporting is too unspecific to be useful?

Are you sure that you want the initial/expected buffer contents in their
own files? Such a split between the inputs and the invocation of the
code under test seems cumbersome. I guess if your inputs/expected
outputs are complex enough, moving them out into files can make sense.
Can you point me to some examples where you think the data is too
complex to be in-line?

One limitations of my approach is that we really need some kind of
"with-test-buffer" rather than "with-temp-buffer" that keeps the buffer
around for inspection if the test fails, but this is item 2.) on your
list already. Another limitation is that the explainer for `equal' is
not as pretty as ediff, but OTOH, as soon as you want to compare text
properties, ediff can't help anymore (AFAIK), while there is an
explainer for `ert-equal-including-properties'.

Text properties (e.g., font lock properties) are important to many kinds
of tests. Both ediff and using files are problematic in such cases
because they don't support text properties. Do text properties never
come up in your context?