Real life scenario

The problem I face is the following. I have some integration tests written with TestNG, and I want to share the results of these tests with our partners from the project. I want a test report, that will make everyone get instantly a clear picture of what is the status of the integration. The tests are grouped in few suites/projects and run independently in OSGi environment.

In this post I will present how to achieve point 2 - a custom report on test execution of single test suite. Having this information, you will be able to create much more complex reports (like the one described above).

Elements

We need three things to make it work. First, we need to obtain information about the tests. This is easy if you use TestNG's listeners. The second point is to create reports without cluttering the code with presentation details. This is crucial, especially if you aim at more then one report format (I do - I want reports in plain txt for quick developers check, and visually stunning html reports). The solution here is to use Velocity Templates, that will decouple your code from presentation issues. The last element of the puzzle is to make the reports look nice - in my case, I will use CSS to achieve this result.

Ah, the fourth element of course ! Build script, yes, we need that too. We will use Gradle for this.

So, we know what will be used, now let's put all the elements of the puzzle together.

The reports

Let us decide first, what element should the report contain. I would like them to show me:

number of tests that passed, failed and were skipped,

if there were any failed tests:

there should be a well-visible warning,

names of failed tests methods (along with names of test classes) should be printed

if there were any skipped tests:

names of skipped tests methods (along with names of test classes) should be printed

if all tests passed, there should be a well-visible green tick

Of course it is easy to add other information, for example - date of tests execution, environment (e.g. Java/OS version).

maps of type Map<String, List<String>>; each entry in this map contains the name of the class and list of methods (that failed in case of <code>failedTests or were skipped in case of skippedTests). BTW. As you can see the map-iteration is possible, but rather cumbersome with Velocity.

test_summary.txt.vm

Even if you are new to Velocity, you will easily understand this "code". It is enough to know, that words prefixed with $ are variables (that are retrieved from the VelocityContext) and expression prefixed with # are control statements.

...and why is TestNG dependency in compile group instead of testCompile ? Because TestListener needs it, and TestListener resides in src/main/java. But yes, this is weird - usually TestNG dependency would be in testCompile group.

The final result

To see the final result execute from the command line:

gradle test
gradle report

and then

less build/test-results/test_summary.txt

or

firefox build/test-results/test_summary.html

I wanted both versions to be very readable. That I only need to look at it, and I know instantly what has gone wrong.

As for the HTML reports I don't use them very often. In fact, I usually check testng-results.xml and don't care about rest.
But if I ever go back to HTML reports and have some new ideas, I will surely let you know.

do you have something specific in mind when you say hudson - testng report customization? We have done some modifications (not report customization though) to the Hudson TestNG plugin to get it working according to our organization needs.

Szkolenia

daily quotes

Unit tests have been compared with shining a flashlight into a dark room in search of a monster. Shine the light into the room and then into all the scary corners. It doesn't mean the room is monster free-–just that the monster isn't standing where you've shined your flashlight.