Guava has staggering numbers of unit tests: as of July 2012, the
guava-tests package includes over 286,000 individual test cases. Most
of these are automatically generated, not written by hand, but Guava's
test coverage is extremely thorough, especially for
com.google.common.collect.

How they were generated? What techniques and technologies were used to design and generate them?

I remember seeing a talk by some google dude that touched this subject. No clue about the name though, the talk was at some java convention I think
–
ZaviorNov 1 '12 at 15:08

3

package com.google.common.collect.testing has a lot of classes with "Generator" in their names - making it look like a framework for tests generation. There are also sub-packages with classes documented as "skeletons" or "base classes" for tests...
–
gnatNov 1 '12 at 16:01

1

@gnat Yes, I was sure I've seen it somewhere. com.google.common.collect.testing.features for instance shows tfeatures/constrains a collection class should satisfy, and a test case is a combination of them. This way they can parametrize testing
–
dzieciouNov 1 '12 at 16:44

2 Answers
2

A large part of this mass of tests is for their collection implementations. They've written generic tests that exhaustively test the collection interfaces, and this generates a suite per implementation. See, for example, classes called CollectionAddAllTester, ListIndexOfTester.

This is all backed by a library called testlib, which ships as part of Guava. This is quite generic. It supports writing generic tests for any interface (not just collections). You can specify Features of possible implementations and test those (e.g. if your set is unmodifiable you expect a different outcome from set.add()), and when you run the tests you specify which features your implementation supports.

It's based on JUnit 3, not 4. Normally, you have a class extending TestCase full of methods named testSomething(), and JUnit runs them reflectively. The testlib library hooks into the running of these tests so that the lifecycle looks like this:

For each implementation you want to test

For each (applicable) test method

Create the TestCase instance

Initialise the TestSubjectGenerator – this is the testlib interface that you extend where you actually create the test subject

Reflectively run the test method. During this method, getSubjectGenerator() gives access to the test subject

The key bit is the extra initialisation step that allows them to inject a specific test subject into the generic test case.

I wrote a post on how to write testlib generating suites for your own interfaces.

There are unit test generators. For example, in the .NET world, something like Microsoft Pex could do this.

For example, Microsoft Pex tries based on code analysis all possible values as arguments for a method. Some arguments are expected to let the method throw an exception. Such things can automatically tests created for. Static values like an empty string that is returned in certain cases can also be automatically be tested.