Flaky tests make your life more difficult. You get failure notifications that aren't helpful. You might become numb to failures and miss an actual failure condition. Your changes might get unfairly blamed for causing a flaky test to fail.

Unfortunately, a myriad of factors can make a test flaky. Today, we tackle a simple example: file access from a unit test. Take this function and its test:

What if /leases/gap already exists and contains some data? testCreateGapLease will fail. This is a general problem where preconditions are assumed to be correct. This could just as easily happen by assuming a database contains the proper information (or no information). What if another test that uses that file was running concurrently?

If you really want to test your code using live resources, always check your assumptions. In this case, clearing the file at the start of the test can reduce its brittleness:

Unfortunately, this doesn't completely eliminate the flakiness of our test. If /leases/gap is an NFS path or can be written to by a different test, our test can still fail unexpectedly. It's better for the test to use a unique resource. This can be accomplished with a small refactoring of CreateGapLease:

3 comments
:

What was wrong with "def CreateGapLease(self, lease_path='/leases/gap'):"? Immutable types like strings aren't a risk in default args. My version probably runs slightly faster and will produce much better automatically-extracted docs.

The related trap, which you might have been trying to avoid, looks like this:

def func(x=[]):. x.append(5). return x

(Except without the silly dots to workaround blogger's formatting)

That program only has one list in it which will keep growing every time you call func(). That's where the "x=None .. if x is None: x = []" idiom is important.

I assume FLAGS.test_tmpdir is a randomly-named temporary directory (e.g. created with tempfile.mkdtemp), otherwise you still might have problems if you run the test suite concurrently and both instances try to write to the same file at the same time.