Project musings

Archive for the ‘Test Frameworks’ Category

Flaky user interface tests don’t generally flake a second time unless there is a real problem. Unfortunately the standard unit test runners don’t offer an option to double-check test failures. This can result in a high build failure rate. After being hit with a large number of suddenly flaky UI tests when a new version of Chrome came out we wrote our own test runner to double-check test failures.

It is compatible with NUnit attribute names out of the box but the attribute names can be replaced via command line configuration switches if your tests are marked with different attributes.

Unique features:

– Scooter double checks all test failures/errors by running the test a second time and only counting a failure if the second attempt fails.

– You can attribute an alternative method (via the -afta command line switch, e.g. -afta=AfterEachFailedTestAttribute) to be called after each test failure. This can be used for capturing screenshots, test failure notification, etc.

– You can also create TestSuiteSetUpAttribute and TestSuiteTearDownAttribute (or equivalent) attributes and make a method for each to perform an action at the beginning and end of the test run respectively.

– Scooter can optionally run the tests in parallel (-parallel) and/or shuffled (-shuffle).

Usage:

Replace the call to NUnit console in your build script with one that calls Scooter.Console.exe instead.
All TestFixtures in your assemblies will be run unless marked with an Ignore or Explicit attributes.

License: MIT License.

Advertisements

Like this:

We finally decided to do something about the time we spend waiting for the pre-check-in build to run. Our check-in dance starts with running a rake script to compile our code, run the associated unit and integration tests, and create deployment Zip files. This entire process takes ~96 seconds on an Intel Core i7-2600K CPU as follows:

Compile – 21 seconds

Run tests – 58 seconds

Build Zip files – 17 seconds

NUnit is clearly the biggest time consumer. A quick check of the task manager while the tests are running shows the following.

Like this:

While reading Karl Seguin’s post entitled Stop Using Mocks I was reminded of a pattern we use at work to reduce test tax. Test tax is the time you spend fixing “broken tests” that result from tests, and in particular mocks, that know too much about your implementation.

This method has calls to two injected tools, a branch after each, and the second is dependent on the output of the first. Tests for this kind of method can become brittle fast. You might easily find yourself fixing half a dozen “broken tests” that are specific to the second injected tool because you changed the method prototype of the call to the first in a way that wasn’t accounted for in the tests. And this is a simple method. Ugh!

We’ve tried a number of different patterns to reduce this test tax. Our current best solution is to rewrite this kind of method into what we call a coordinator. Coordinator methods have no branches and are tested by integration tests, that means real objects instead of mock objects. The process they are coordinating is broken down into a series of steps that can be unit tested in isolation. The step methods do not generally appear on the interface of the coordinating class.

Karl’s method might be implemented using the coordinator pattern as follows:

This lets us write detailed tests around GetUserByUserName and GetUserIfPasswordMatches without worrying about a change in the implementation of one method unexpectedly breaking the tests for the other. At the integration test level we can verify broader behaviors and interactions with real objects.

When my cursor is on one of the test actions and I hit alt-enter Resharper is usually smart enough to suggest creating a method.

Other times, however, Resharper doesn’t offer the option we need.

My solution to this problem is a Visual Studio macro that creates a void parameterless method for the “word” the cursor is on. This necessitates determining where the new method should be created e.g. top of the file, bottom of the file, just before the method, just after the method. I like to keep the new method close to the test when I’m first creating it so that means above or below the method. Initially I tried it above the method but learned that in my workflow I expect the new method to be added below the test. So how do we do that? We’ll write a Visual Studio macro to do the work.

First we need to copy the proposed method name. We’ll grab the word under the cursor and put it in a variable. Also, since we don’t want to affect the user’s copy-paste buffer we’ll do it within an undo context.

DTE.UndoContext.Open("AddBddMethod")
DTE.ExecuteCommand("Edit.SelectCurrentWord")
Dim name As String
name = DTE.ActiveDocument.Selection.Text

Next to move to the end of the test method we’ll search for ‘}’ and close the find dialog when we’re done.

Like this:

I’ve been using FluentAssert for my test-driven development because it abstracts away the test details leaving only a highly readable test. I also find that it enforces a certain level of separation of concerns and moistness in the tests without spreading portions of the tests through an inheritance stack. For example:

Today I was reading about features of the Factor language, specifically the part about naming code not values and that it is stack based instead of function based, and it occurred to me that a DSL that has no parameters is a lot like a contactenative language except for the extra visual bits like . and () joining the method calls together. This started me to thinking how I might be able to further reduce the ceremonial aspects of the test framework.

One nice thing about a DSL is it can be coded to be context sensitive, meaning it might not let you put an assertion call (e.g. Should()) before an arranging call (e.g. With()). A lower ceremony framework wouldn’t necessarily offer that level of protection but in return we might get tests that are even easier to read and that don’t have an execution command appended. That’s one thing that bothers me about the current framework, the command of execution, .Verify(), is required at the end of each test chain in order to fire off the framework test runner. If you leave it off the NUnit test runs but the BDD test doesn’t and you get a false green. I wondered if I could address that annoyance as well as make something more concatenative. Here’s the result:

Instead of chaining methods with Action parameters we’re passing all the Actions as a params array. The DSL methods .When(part_A) .Should(part_B) etc. are simply replaced with their parameters part_A, part_B but we can change the Action names to include the When, Should etc. in order to retain readability in the test and its output. The call to Verify becomes the only framework method we call and we can give it the test Actions to run in the order we want to run them instead of having to give the Action for the method under test (When()) first – this is about as close as I think I can get to a Forth style Word command line in C#.

I’ve integrated this variation with the existing test result reporting so if you start your Action names with known keywords you’ll continue to get uppercased keyword output, e.g.:

WITH a null string
WITH trim set to false
WHEN asked if a string is null or empty
SHOULD return true - FAILED
Expected: True
But was: False

I’ve added this as an alternative DSL path in the FluentAssert framework on GitHub if you want to try it out.

Enjoy!

Like this:

The gist of BDD (Behavior Driven Develoment) is being able to write tests in language that a non-programmer can read and verify. Until now I’ve used a style that I adapted from J P Boodhoo that looks like the following:

An outer class for all tests of a particular class

public class CollectionTests
{
}

that contains TestFixture classes for each action (method) on the class under test

This makes tests that are highly readable in NUnit with test run results that can be coerced into a Word or Excel report. The main problem is the bodies of the Setup TearDown and particularly the Test methods get so cluttered with Mock expectations and other goo that even developers can have a hard time looking at them and verifying their correctness not to mention fixing breakage.

The first thing is we don’t want to have to inherit from a base class to get access to Given() so that implies some kind of starter class. It could be an extension method, or a class that has to be new’d up, but I prefer a static class that starts a DSL for type T.

that the things passed to With(), When() and Should() don’t have parentheses… and they aren’t lambdas either. If we want this to exactly match the example the only things left are Func and Action. In the case of With() we’re going to want to take a T and make a change to it but we don’t need to return anything so that implies Action. Based on the example we also want to be able to call With() multiple times so internally we’ll store the Actions in a List:

Now the example shows only one Should() clause but we’ll want that to be repeatable too (although in practice I generally only have one assertion per test method) so that means the implementation is just a variation on With()

Does that look familiar? It should. We’ve just written a wrapper around the AAA (Arrange Act Assert) pattern that lets us abstract all the details out of the actual test.

Now to use it we write the test in language as close to business-speak as possible. The first difference is we’re going to be testing a Repository, not a Person (because we’re separating concerns) so the test comes out as:

Now that we can see the final product the .For<T> part is a bit icky to look at and certainly wouldn’t be useful to a business person. Let’s clean that up a bit. We can get the T by induction if we just pass the object so we’ll add Given() to Test too:

This framework has the side benefit of forcing us to break up our Actions into composable chunks and keeps our test classes moist and maintainable. Now one could go so far as to split the parts among the [Setup] (For, Given, and When) [Test] (Should) and [TearDown] (Verify) but I think that takes us away from the goal – tests that are so easy to read that a non-developer can understand and verify their correctness and detect/suggest missing test cases.

One more thing to think about: Given a project where many tests are written in this style it wouldn’t be too difficult to write a tool to crawl the source code for methods marked [Test] that start with the Test.Given() construct, grab their innards and drop them all in some kind of document that could be reviewed by an analyst. Hmmm.