RegTest: Simple IF Regression Tester

RegTest is a very simple script for writing IF unit tests. You write down a list of commands and the output you want from each one. RegTest will run the list through your game, and check for that output.

RegTest can work with any interpreter which uses stdin/stdout, such as DumbFrotz or Glulxe/CheapGlk. (But see "Limitations", below.)

The Test File

All of what RegTest does is defined by a test file. The easiest way to explain it is to paste one in. (With a soupçon of syntax coloring, for documentation's sake.)

# advent-regtest: test script for regtest.py# For a full description, see <http://eblong.com/zarf/plotex/regtest.html>
** game: /Users/zarf/Documents/IF/Advent.ulx
** interpreter: /Users/zarf/bin/glulxec -q
* south-from-start# A simple, one-command test.> south
You are in a valley in the forest beside a stream tumbling along a rocky bed.
* in-well-house# Test the opening text, followed by two commands. Lines starting# with "!" are negated; lines starting with "/" are regular expressions.
Welcome to Adventure!
Crowther
Woods
> go east
There is tasty food here.
some keys
!grue
> get all
/b[aeiou]ttle.*water
!/^Taken
* test-that-fails# All three of the tests in this run will fail.> go east
There is a bucket of cheese here.
/[xqz]
! Inside Building
* remglk-test# This test will only work if your interpreter uses the RemGlk library,# and regtest is invoked with the --rem option. Otherwise, you'll see# three test failures.> go east
{status} Inside Building
spring
> help
{status} About Adventure
>{char} N>{char} N>{char}
The probabilities are as in the original game.
>{char} 32>{char} Q
{status} Score
You are inside a building
> get food
Taken.

The first two lines are comments. Lines beginning with "#", and blank lines, are ignored.

The next two lines (beginning with "**") define test parameters -- the location of the game file and interpreter. The game will be the Glulx version of Adventure (compiled with Inform 6). The interpreter will be Glulxe/CheapGlk. I've defined pathnames in my computer's filesystem; you'd want to change those, of course. (You can also supply these values from the command line.)

A line beginning with "** precommand:" is an extra command that will be stuck onto the beginning of every test defined in the file.

A line beginning with "** checkclass:" specifies a (Python) file containing extra check classes. I won't get into the details here, but see this sample file.

The rest of the test file is a set of tests. Each test is a separate run through the game. A test contains a sequence of commands. A command can contain various checks, validating the output of that command.

(All the "**" lines should appear before the tests begin.) (Okay, you could customize the game file or interpreter for a specific test if you really wanted. But why?) (This is a rhetorical question.)

The line "* south-from-start" defines the beginning of the first test. south-from-start is the test name. (You can name tests anything you want; it's just a convenient label.)

This test contains just one command -- south. The next line is a check: RegTest will search the command's output for this line. It's the room description for the room to the south, obviously.

The second test is called in-well-house. Here we start by performing some checks on the banner text of the game. (Note that this test is a fresh start; the previous "south" command was in a different run.) RegTest verifies that "Welcome to Adventure!" occurs somewhere in the game's initial output. Then it looks for "Crowther" and "Woods", which also occur. (These aren't complete lines, but that's fine -- the check line just has to occur somewhere in one of the paragraphs that begin the game. The two name tests happen to occur in the same line; that's fine too.)

After the initial text, we go east. We're applying three different checks to the output of "go east". RegTest verifies that "There is tasty food here." and "some keys" both occur. (Remember, we're looking only at the output of the latest command, not the entire transcript.)

A check line starting with "!" is negated: RegTest verifies that none of the output contains the word grue. Which is good, because there are no grues in Colossal Cave. You can also use "{invert}" as the line prefix.

These are independent checks; order doesn't matter. (The line about the keys actually occurs before the one about the food.)

The idea is that you don't usually want to verify every single character of your game output. During development, you're going to be changing descriptions, adding objects, and so on. But you still might want to write a test sequence for particular actions. By checking only for the important bits of each response, you don't have to fix the test every time a room description or timer event changes.

The next command demonstrates regular expressions. A check line that begins with "/" is matched as a regular expression. (See the Python documentation for the syntax of regular expressions.) Here we have a (contrived) regex which matches the output line "stream: The bottle is now full of water."

A line starting with "!/" is, unsurprisingly, a negated regex check. The line "!/^Taken" verifies that no line of the output begins with the word Taken. (The word occurs within several lines, but not at the beginning of any.)

The last test, remglk-test, is its own crazy thing. We will discuss it momentarily.

Specify the location of the game file. (This overrides the **game: line in the test script.)

-i, --interpreter:

Specify the location of the interpreter. (This overrides the **interpreter: line in the test script.)

-l, --list:

Do not run the tests; just list them.

-p, --precommand:

Specify a precommand, which will be run before every test. You can give several precommands. (These add to the **precommand: lines in the test script.)

-c, --cc:

Specify a file of custom check classes. (Adds to the **checkclass: lines in the test script.)

-r, --rem:

The interpreter uses RemGlk (JSON) format.

--vital:

Abort any test run at the first error.

-v, --verbose:

Display the game transcripts as they run.

Partial Tests

Sometimes you want to wrap up a sequence of commands as a "macro", to be invoked in several different tests.

To do this, add a command line like this:

>{include} TESTNAME

You can name any other test in the file. Its commands (and checks) will be executed at this point in your test.

(No space between the ">" and the "{". Checks after an >{include} line are meaningless; they are ignored.)

You typically won't want a subtest to be invoked by itself. (The player won't start in the right place, so the subtest's checks will fail.) To make this convenient, give the subtest a name beginning with "-" or "_". Such tests will not be run when you invoke RegTest in all-tests mode (or with "*").

Limitations on Cheap Mode

Normally, RegTest handles IF output in a very simplistic way. Because the stdin/stdout model has no facility for a status line, there's no way to test the status line's contents. Also, RegTest will only work with a game that abides by these rules:

The prompt must always be ">" at the beginning of a line.

In particular, Inform's "if the player consents" (yes/no) questions will confuse RegTest -- it won't recognize them as input requests. The same goes for menu-based input.

">" at the beginning of a line must always be a prompt.

If your game prints ">" at the beginning of a line, even if text follows, RegTest will think it is an input request and fire the next command.

This is not very flexible. Can we do better? We can -- but we'll require a special interpreter.

RemGlk Mode

If your interpreter is compiled with the RemGlk library, it will output the full display state of the game, structured as a JSON file. This means that RegTest can see the contents of the status line, and handle more complex I/O requests.

(The JSON format for the game's output, and its input, is described in this document. But you don't need to understand the details to use RegTest.)

The last test in the test file, remglk-test, makes use of this feature. To make it work, compile Glulxe and RemGlk, and then change the **interpreter line to refer to the new interpreter binary. You can then run RegTest with the --rem option. (This tells RegTest to expect JSON-formatted output, rather than plain text.)

python regtest.py --rem TESTFILE

The remglk-test will now succeed. (test-that-fails will still throw its three errors.)

The test demonstrates two special features: character input and status line output. We enter the game menus by typing "help"; we then navigate to one of the menu options and trigger it. We test the option's output -- this is the "How authentic is this edition?" text. Then we hit space (ASCII 32) to return to the menu, then "Q" to return to the game. We can then proceed with game commands as before.

> help
{status} About Adventure
>{char} N

These features are signified by lines with {curly brace tags}, as shown above.

Dictionary of Inputs

When writing these input forms in a test, do not put any whitespace between the ">" and "{" characters. An input line like "> {foo}" is treated as regular line input, entering the string "{foo}".

> text

Regular line input.

>{char} X

>{char} escape

>{char} 123

>{char} 0x1F0

Character (keystroke) input. A single character stands for itself. Standard keystroke names (left, escape, etc) are accepted, as are ASCII or Unicode code as decimal or hexadecimal. If you do not provide a value, RegTest assumes a Return keystroke.

Note that in RemGlk mode, RegTest is able to tell whether the game is expecting line or character input. It will report an error if your script offers the wrong one.

>{timer}

Timer input. RegTest does not really perform a delay for timer events; it just tells RemGlk that the timer has fired. (When running a game which uses timer events, you should pass the -support timer option to RemGlk.)

>{hyperlink} 123

Hyperlink input. The value should be a (decimal) integer containing a link value. (When running a game which uses timer events, you should pass the -support hyperlinks option to RemGlk.)

>{fileref_prompt} savefile

The response to a file prompt (save, restore, transcript, etc). The line should contain a simple filename (no directory, no suffix).

>{arrange} 640

>{arrange} 640 480

A window arrangement event. The virtual game window will be resized to the given pixel width, or width and height. (The virtual monospace font is assumed to be 10x12 pixels.)

>{refresh}

A browser refresh event. This simulates the situation where GlkOte is reconnecting to the game and requires the entire display state to be retransmitted.

>{debug} backtrace

A debug command. The value will be sent to the game (or, usually, to the interpreter) as a debuginput event. Note that RegTest cannot currently check debug output.

>{include} testname

Performs all the commands and checks in the named test.

Dictionary of Check Modifiers

You may put any of these prefixes before a check. They may be combined freely (except for {status} and {graphics}).

!CHECK...

{invert} CHECK...

Invert the sense of the check -- test that it is false.

{vital} CHECK...

If the check fails, end the test run immediately.

{status} CHECK...

Test the contents of the status window, rather than the story window.

{graphics} CHECK...

Test the contents of the graphics window, rather than the story window.

RegTest currently assumes that there is no more than one status (textgrid) window and no more than one graphics window.

Dictionary of Checks

Any of these checks may be combined with the modifiers listed above.

text

Check whether the text appears anywhere in any line of the output.

/regex

Check the given regular expression matches any line of the output.

{count=5} text

Check whether the text appears at least the given number of times in the output. (In this example, we check whether the string "text" appears 5 or more times. Could be in separate lines of the output or all in the same line.)

{hyperlink=123} text

Check whether there is a hyperlink with the given (decimal) value, which also contains the given text.

{image=5}

{image=5 x=100 y=200 width=32 height=32}

{image=5 width=32 height=32 alignment=inlineup}

Check whether there is an image with the given number. The rest of the modifiers are all optional; they are checked if present.

Remember that images can appear in graphics windows (with x, y, width, and height values) or in the story window (with alignment, width, and height values).

{json key:'value' key2:123 key3:true}

Check whether a JSON stanza appears as a span in the output. This allows you to check the raw JSON output for special forms. For example, this checks for the operation of drawing a 10x10 pixel red square in a graphics window:

{graphics} {json special:'fill' color:'#FF0000' width:10 height:10}

Note that we use the {graphics} modifier (see above) on a {json} check.