One of the great advantages of moving the code hosting in GNOME
to GitLab is the ability to run per-project, per-branch, and
per-merge request continuous integration pipelines. While we’ve had a CI
pipeline for the whole of GNOME since 2012, it is limited to
the master branch of everything, so it only helps catching build issues
post-merge. Additionally, we haven’t been able to run test suites on
Continuous since early 2016.

Being able to run your test suite is, of course, great—assuming you do have
a test suite, and you’re good at keeping it working; gating all merge
requests on whether your CI pipeline passes or fails is incredibly powerful,
as it not only keeps your from unknowingly merging broken code, but it also
nudges you in the direction of never pushing commits to the master branch.
The downside is that it lacks nuance; if your test suite is composed of
hundreds of tests you need a way to know at a glance which ones failed.
Going through the job log is kind of crude, and it’s easy to miss things.

Luckily for us, GitLab has the ability to create a cover
report for your test suite results, and present it on the
merge request summary, if you generate an XML report and tell the CI
machinery where you put it:

artifacts:reports:junit:-"${CI_PROJECT_DIR}/_build/report.xml"

Sadly, the XML format chosen by GitLab is the one generated by
JUnit, and we aren’t really writing Java classes. The JUnit XML
format is woefully underdocumented, with only an unofficial
breakdown of the entities and structure available. On top of
that, since JUnit’s XML format is undocumented, GitLab has its own
quirks in how it parses it.

Okay, assuming we have nailed down the output, how about the input? Since
we’re using Meson on various projects, we can rely on machine
parseable logs for the test suite log. Unfortunately, Meson
currently outputs something that is not really valid JSON—you have to break
the log into separate lines, and parse each line into a JSON object, which
is somewhat less than optimal. Hopefully future versions of Meson will
generate an actual JSON file, and reduce the overhead in the tooling
consuming Meson files.

Nevertheless, after an afternoon of figuring out Meson’s output, and reverse
engineering the JUnit XML format and the GitLab JUnit parser, I managed to
write a simple script that translates Meson’s
testlog.json file into a JUnit XML report that you can use with GitLab
after you ran the test suite in your CI pipeline. For instance, this is what
GTK does:

Some assembly required; those are XFAIL reftests, but JUnit doesn’t understand the concept

The JUnit cover report in GitLab is only shown inside the merge request
summary, so it’s not entirely useful if you’re developing in a branch
without opening an MR immediately after you push to the repository. I prefer
working on feature branches and getting the CI to run on my changes without
necessarily having to care about opening the MR until my work is ready for
review—especially since GitLab is not a speed demon when it comes to MRs
with lots of rebases/fixup commits in them. Having a summary of the test
suite results in that case is still useful, so I wrote a small conversion
script that takes the testlog.json and turns it into
an HTML page, with a bit of Jinja templating thrown into it to
avoid hardcoding the whole thing into string chunks. Like the JUnit
generator above, we can call the HTML generator right after running the test suite:

And GitLab will store it for us, so that we can download it or view it in
the web UI.

There are additional improvements that can be made. For instance, the
reftests test suite in GTK generates images, and we’re already uploading
them as artifacts; since the image names are stable and determined by the
test name, we can create a link to them in the HTML report itself, so we can
show the result of the failed tests. With some more fancy HTML, CSS, and
JavaScript, we could have a nicer output, with collapsible sections hiding
the full console log. If we had a place to upload test results from multiple
pipelines, we could even graph the trends in the test suite on a particular
branch, and track our improvements.

All of this is, of course, not incredibly novel; nevertheless, the network
effect of having a build system in Meson that lends itself to integration
with additional tooling, and a code hosting infrastructure with native CI
capabilities in GitLab, allows us to achieve really cool results with
minimal glue code.