Testing our work

Posted on June 28, 2017
by Adam Wespiser

Testing shows the presence, not the absence of bugsDijkstra

Testing w/ Haskell

Within dynamically typed language like Scheme, we lose the safety of Haskell’s type system and need an alternative guaranty of behavior. This chapter is about writing tests to ensure our Scheme is behaving as we expect. Fortunately, testing can be easily integrated into a Stack project, and Haskell’s many frameworks satisfy a multitude of testing requirements. We will be looking at a few testing options, and implementing both HSpec, and Tasty.Golden. There are two files for testing, test-hs/Spec.hs which contains the parser tests and golden file tests, and test-hs/Golden.hs, that contains just the golden file tests.

Haskell Testing Frameworks

Haskell has a few good testing frameworks, here are a few of them, and what they do:HUnit. A Simple embedded DSL for unit testing.HSpec is a straightfoward testing framework, and gives great mileage for its complexity. HSpec is inspired by RSpec a testing library for Ruby, and the two show remarkable similarity.QuickCheck. Tests properties of a program using randomly generated values.Tasty Test framework that includes HSpec, Quickcheck, HUnit, and SmallChcek, as well as others, including Golden, which we will use for testing against a value within a file.

This enables stack test and stack build --test to automatically run the HSpec tests found in test-hs/Spec.hs Running one of these commands will build the project, run the tests, and show the output. Phew! All tests pass! (let me know if they don’t)

Now we can run stack test --test-golden to run the tests from test-hs/Golden.hs Running stack test will perform both test suites. Let’s look further into how testing is done, and the libraries used.

HSpec Setup

HSpec’s strength is its simplicity, and ability to compare the results of arbitrary functions against a known output. We will use it to test internal components of our Scheme, particularly the parser, which contains a depth of logic orthogonal to the rest of the code base.
First, let’s take a look at the general form of an HSpec test.

describe gives a name to the block of tests, which is printed out when the tests are run.

it sets a specific test.

shouldBe states that a specific expression matches the test expression.

HSpec Tests

Two internal aspects of our Scheme will be tested in test-hs/Spec.hs: The parser, and evaluation. These two features lend themselves easily to testing, and together, cover ensure functionality meets expectations.
Another view is that these tests allow us to modify the project without changing the features we worked so hard to implement, test driven development (TDD). The ./test folder in our project contains the Scheme expressions run during the tests. Besides files containing expressions, we can also specify expressions as T.Text, and define blocks without loading the standard library. All of the parsing logic, and evaluation of simple expressions, special forms, and features like lexical scope are included in the scheme expressions found in the test folder.

The first set of tests ensures text is properly parsed into LispVal using readExpr. To organize this set of tests, the hspec function is used, along with describe to give the set of tests an suitable description. Many constructions of LispVal are tested, and here were divide that list into S-Expression and non-S-Expression values for simpler testing.

Alright, on to evaluation. Our task here is ensuring there are no errors, bugs, or unspecified behavior in our Scheme… If there were only a way to incorporate a system that protects us from invalid programs… Type systems be damned! We are all that is Scheme!
To test evaluation, we are going to either: read and parse from a file or inline text, then run with or without loading the standard library. This way, we have flexibility over testing conditions, especially considering the standard library will be subject to the majority of iterative testing and revision efforts.

What’s troublesome here is the use of unsafePerformIO to read file contents and shed the IO monad. Stepping back, we are coding a test within a specific file, evaluating it with or without the standard library, then comparing it to a value compiled into the testing file. If we can admit HSpec is good at testing internals like the parser, its also fair to say its not great at this process of “golden tests” for our Scheme language. Fortunately there is a better way that allows us to run a test Scheme file and compare the result against a ‘golden’ value in a stored file!

Tasty Golden Tests

goldenVsString ::TestName-- ^ test name-> FilePath -- ^ path to the «golden» file (the file that contains correct output)->IOLBS.ByteString-- ^ action that returns a string->TestTree-- ^ the test verifies that the returned string is the same as the golden file contents

This allows us to run the tests located in ./test and compare the results to the likewise named files in ./test/ans.