Recent Updates

For a long time I have been on the lookout for a ready-to-use R-based API for testing ML algorithms. My ideal “tool” would be an R-based API for the game “go” (“weiqi” in Chinese and “baduk” in Korean), but I found none so far.

Since writing a rules engine is quite time-consuming, I would not venture developing one until recently when I stumbled upon the game “2048.” Many people use this simple game to benchmark their machine learning algorithms; and YouTube has so many machine learning demos using “2048” that I won’t even list them here. To my surprise, all the videos I watched seem to use a java script based front-end for testing. Using a java script connection seemed like an over-complication for me including an additional bottleneck.

So after spending an hour building my own R-based version of “2048,” I decided to see what other options were available; and I found this nice implementation in C: https://github.com/mevdschee/2048.c by Maurits van der Schee. Porting from C saved me a lot of time. Many thanks to the original author.

I added a simple API for attaching the game code to a ML algorithm. I will just make a note that each game may be run in its own R environment, which allows for easy set up of parallel computing using the standard “foreach.” Other than that, the whole code is a little over 400 lines, so everything about its usage should be self-explanatory.

The program could be run in an interactive text mode as well (using `main_interactive()`). However, I found no way to port text coloring to the console of RStudio. Still, as this code is meant to be played by machines rather than human users, I suppose, I achieved my goal.

This is a tool for code fault analysis. I built it to automate the most boring part of my debugging process.

The package automates test setup and logging and visualizes function exit states in a way that simplifies identification of root causes of software defects. Fuzzing is implemented by random generation of input parameters as shown in a demo below. Finally, even though there was no goal to make this tool as another unit testing package, one can use it as such, as this tool should potentially assure 100% code coverage with minimal effort.

The package tests all possible combinations of input parameters and produces statistics and visuals. If you have some specific requirements, you can simply build a wrapper function that catches output you are interested in and generates an error if a required condition is not met. Then submit your wrapper function for testing. You can even compare current output values against values recorded in a log during a ‘reference’ test run, effectively making a comprehensive unit test.

===LOG OMITTED===
Fuzztest: Argument-Option Combination Results
===================================================
ARG~OPT Arg Name PASS FAIL FAIL%
---------------------------------------------------
1 ~ 1 x 5 7 58.3
2 ~ 1 y 5 7 58.3
3 ~ 1 option 4 0 0.0
3 ~ 2 option 1 3 75.0
3 ~ 3 option 0 4 100.0
4 ~ 1 suboption 2 1 33.3
4 ~ 2 suboption 1 2 66.7
4 ~ 3 suboption 1 2 66.7
4 ~ 4 suboption 1 2 66.7
===================================================
Fuzztest: Summary
========================================================================
Arg Name Failure Rate Contribution, % (Max - Min)
------------------------------------------------------------------------
x 0.0 ' '
y 0.0 ' '
option 100.0 '**************************************************'
suboption 33.3 '***************** '
========================================================================
The summary shows that the argument 'option' explains the most
variability in the outcome. So let's concentrate on the arg. 'option.'
The detailed statistics table shows that most failures occur when
value #3 is selected within the argument 'option'. At the same time,
a test log (omitted here) shows that the types of errors are mixed.
For now, however, let's assume that fixing the bugs related to
control flow is more important.
The following three graphs will demonstrate how the data above
can be represented visually. Notice that some lines are grouped
when they intersect vertical axes. The groups correspond to specific
options and are ordered from the bottom of the chart to the top:
i.e. the fist grouping of lines at axis 'suboption' (at the bottom)
corresponds to value 'a', the next one up is suboption 'b', and so on.
In case an argument has only one value in the test, the whole
group of lines will be evenly spread from the bottom to the top of
the chart, as is the case for arguments 'x' and 'y'.
* All test cases:
One can also selectively display only passing or failing tests
as will be shown next.
* Only 'passing' test cases:
* Only 'failing' test cases:
Let's assume all the control flow related bugs discussed above
are fixed now. To make this assumption to "work" during testing
we will simply choose a combination of options that will not
cause the demo function to produce 'fail' states shown above.
Such a combination could be {x=0, y=0, option='a', suboption='a'}.
Now we will concentrate on the numeric part of the test.
There are two main testing approaches:
1. Create an evenly spaced sequence of values for each parameter
(x and y) from lowest to highest and let the argument set generator
combine and test these values. This approach has an advantage
for more intuitive visualization as sequences of values
for testing will be aligned with the vertical axis. For example,
if we create a test sequence [-10;+10] for argument 'x',
visualized test results will list those from 'Min' to 'Max'.
So finding simple linear dependencies that cause errors will
be easier as it will be easier than when using a random set
of values (below).
2. Generate random parameters for selected arguments and let the test
framework test all possible parameter combinations.
The First Approach: Ordered Test Sequences

After reading this post [http://yihui.name/en/2013/09/testing-r-packages/], I realized that saving reference values to compare against them the output after code is modified does not allow TDD, or test driven development. So the tests will always “drag behind” the development process.

I am currently using option #3. However, there are obvious shortcomings of this approach in large projects. Since I am using R as well as other languages, naturally, my choice falls on RUnit (xUnit framework) as multiple languages use this format and this fact will make life easier in the long run.

Key points about the testing workflow:

install the package

test the package

testing in development mode is a separate matter and won’t be my primary concern

“package_root/inst/unitTests” contains a file with the primary testing suite builder code {suit_builder example} and test code files {test files examples}. “unitTests” folder will be moved into the package root folder after installation and will become accessible to the ‘laucher’ R code sitting in the “package_root/tests” folder.