Introducing Exemplar for Automated Samples Testing

This post introduces a new library called Exemplar.
The goal of Exemplar is to ensure that users get outputs that you expect them to see.
It handles sample discovery, normalization (semantically equivalent results, perhaps from different environments), and flexible output verification.
It invokes any command-line tool in the environment to be invoked. You can also invoke curl, for example, to verify service API responses.

Gradle uses this library to verify examples in docs and guides, and remove boilerplate from integration tests.

Exemplar can be configured using a JUnit test runner (recommended) or using its APIs.
See examples below and in the Exemplar GitHub repo.

Use cases for Exemplar

It’s important that documentation samples are accurate.
Imagine the frustration, trying to learn something new and having the examples fail to work.

We think Exemplar works best as a documentation checking mechanism.
It is not meant to substitute other forms of integration testing however.

Exemplar is unique because it allows discovery of samples embedded in structured documents, as well as OutputNormalizers (bundled and custom ones), conveniences for using sample projects in “more traditional” integration tests (via @UsesSample("path/to/sample")), and allows samples to be programmatically modified (for extra environment setup, perhaps) before execution (SampleModifier).

Exemplar is focused on logging and exit code outputs, and verifying other side effects is cumbersome (checking created files may require extra commands).
Furthermore, although Exemplar can be used for any tool, it only has APIs for JVM projects.

Sample project testing primer

The sample-discovery library in Exemplar will consider any directory containing a *.sample.conf file under the samples root as a sample project.

A sample config file is written in HOCON and tells sample-check how to run the sample project.
It can be simple:

Adopt and contribute

Exemplar is at version 0.6 at the time of writing this post, which means that the APIs may change before v1.0.
That said, given that the API has gone through some revisions already and has been adopted by Gradle, breaking changes are unlikely before version 1.0.

Here are some of the things that might be good expansions of the library, and I’d love your help if you’re eager to file ideas or submit pull requests to the gradle/exemplar GitHub repo.

More standard normalizers such as date, time, or duration normalization

Generate structured metadata for search indexing

Allow expected output to be inlined in sample config files

Samples discovery embedded in Markdown or other documentation formats (#2938)

Please reach out to me (@eriwen on Twitter) if you’d like guidance adopting or contributing to this project.