Testing gen_fsm – Part 6 ATG engine – Implementation

Finally, after all this thinking and all the discussing with ourselves, the moment of implementation is here. Of course, I have been writing this up during the ever so minimal spare time, and will present the final result for your pleasure, and also walk you through the interesting bits and pieces while commenting it. Enjoy!

Overview

The general idea is that the test module will include the atg_parsetrans as a parse transformation, and the state representation of the gen_fsm (often a record). The test module will then define a set of 3-tuples which act as a kind of Hoare-triples. The atg_machine is then used for using the test module-defined rules and auxiliary functions to randomly test sequences of transitions through the gen_fsm.

In detail – atg_parsetrans.erl

The atg_parsetrans module is the parse_transform library that your test-module should import (for your convenience). atg_parsetrans will convert the kind-of Hoare Triples to an atg_machine friendly format. The transformation of your test module will then be roughly

Encase all expressions in fun’s

Store a string representation of the original expression, used for failure reports

Put the fun encased Expr and the String representation into internal atg record

In order to fully understand this, the prerequisite knowledge is ‘The Abstract Format’. The key parts to know is that the entry-point of this module is the exported one (of course) and that it works on a list of abstract format terms. I then let each function clause of my transform/1 function take care of each specific thing that interests me.

Also, there is a specific function for traversing the list of 3-tuples holding the rules. That function is interesting as it shows how I rewrite the code to be fun()-encapsulated and stringified. For this, I use erl_prettypr a very handy Abstract Form to String library, erl_scan the Erlang tokenizer (I hope you understand parsers and programming languages), and the erl_parse for generating replacement Abstract Form code.

As can be seen, this function replaces each tuple element Expr with a #rule_fun record where the function field is bound to a fun encasing a parse-transformed version of the original Expr, and the string field is bound to the original un-parse-transformed Expr.

In detail – atg_machine.erl

The atg_machine is the whole .. yeah, machinery. Pumping through the different #rule_fun records, finding the matching ones, selecting one at random and executing. The coarse steps of the atg_machine are

Check if end condition is met

Check if maximum iterations are met

Find matching set of accessible Rules

Pick one Rule at random from accessible rules

Execute the program of the picked Rule

Check postcondition of the picked Rule

One specially interesting thing is that if no rule can be applied before any endcondition is met, it will be considered a failure.

Some interesting things to discuss could be that this machinery does not need parse_transforming and never needs to know the internal format of the gen_fsm. So, it’s gen_fsm implementation agnostic. It will run until the first thing happens of [condition failure, end condition success, 100 iterations passed].

Other than that, I believe the code is fairly self explanatory. Of course, be welcome to ask in the comments.

A test module

A user supplied test module (call it what you want, for example: demo.erl) should define that parse-transformation is to be done with the atg_parsetrans module.

-compile({parse_transform, atg_parsetrans}).

The test module must also define the internally used state record of the gen_fsm if such a record is used and it will be very handy if the following functions are exported from the test module

-export([rules/0,is_done/0,start/0]).

why are these functions so handy, and what should they contain?

The rules() function is an obvious way to pass the Rule-list to the machinery, thus, this function is defined to return {‘$RULES’, [….]} where the dots ‘.’ in the list are the 3-tuples. Something like this

The is_done/0 function will serve as the end predicate for the machinery, so it will know when it’s “done”, can be set to return false if MAX_ITERATIONS of tests are wanted.
Example could be

is_done() -> '$STATE'#state.deal_done == true.

And the final export start/0 will serve as the function that return the value that will be bound for the gen_fsm accessing.
Example

start() ->
{ok,Pid} = tradepost:start_link(),
Pid.

Thus, we understand, that the majority of the work lies in writing the rules/0 function and all the 3-tuples defining the rules. The programmer will have to tip in some extra work into the predicates. Now for the example test module.

Usage examples

My own test module (demo.erl) will be posted below, together with the success output and some error examples when we do some bad modifications to the tradepost.erl code.
First the demo.erl code

There, this is the test module I have for the tradepost.erl module, and here comes the compilation and running. I have already compile atg_parsetrans and atg_machine to ebin/ and therefore add ebin/ to the path when compiling demo.erl

The 11 dots seen show that 11 cycles where run before the is_done/0 predicate returned true. Now, if we wish to see some errors, let’s make a small modification (that actually was a bug I discovered thanks to my own tool) in tradepost.erl

Now, first of, the 8 dots say that 8 cycles where performed, and the Error output says that we failed on the 8th cycle, that is, after 7 completed. The phase of the error was the postcondition of the selected rule. The selected rule is shown on the “Rule:” line. The current state in which we failed is shown after “State:”.

The “Log:” shows a list of all the processed and passed rules, most recent nearest the top of the screen. This is also true for the “States:” output, which serves as a log for us. The internal process dictionary bindings for ‘$ID’ and ‘$PROGRAM_RESULT’ are also shown.
In short, almost everything we need for debugging is here.

Now for a really terrifyingly delicious comment! I had to run this several (6 times) before this error was shown.

Why? Because the testing is randomised. And some of the corner cases will thus not be triggered easily during short runs. Let us undo the faulty change and introduce an easier one into the program. Let’s forget that buyer inserts cash and not item. (Line 42, change cash to item : and recompile!)

Bam! Reading the error reason, it becomes apparent that something is not working in the cash insertion! the cash in the state does not match ‘$PROGRAM_RESULT’ which was 100! Looking at the State, it’s undefined. Bummer! :)

This was the final part of the home-written ATG, I will presumably put in on git if the demand is high enough.

Roman Shestakov

Wow, this is one of the most interesting erlang related posts I have ever read. Even setting aside valuable testing techics , it gives a great example of how to use parse transformations. Thanks you very much! Is this code available somewhere on github?