Navigation

This document contains information about testing the release candidates that
will ultimately be the next LLVM release. For more information on how to
manage the actual release, please refer to How To Release LLVM To The Public.

This script will check-out, configure and compile LLVM+Clang (+ most add-ons,
like compiler-rt, libcxx, libomp and clang-extra-tools) in
three stages, and will test the final stage.
It’ll have installed the final binaries on the Phase3/Releasei(+Asserts)
directory, and that’s the one you should use for the test-suite and other
external tests.

It should have no new regressions, compared to the previous release or release
candidate. You don’t need to fix all the bugs in the test-suite, since they’re
not necessarily meant to pass on all architectures all the time. This is
due to the nature of the result checking, which relies on direct comparison,
and most of the time, the failures are related to bad output checking, rather
than bad code generation.

If the errors are in LLVM itself, please report every single regression found
as blocker, and all the other bugs as important, but not necessarily blocking
the release to proceed. They can be set as “known failures” and to be
fix on a future date.

Using the Phase3/Release+Asserts/llvmCore-MAJ.MIN-final.install base,
run the test-suite.

If the final phase’s makecheck-all failed, it’s a good idea to also test
the intermediate stages by going on the obj directory and running
makecheck-all to find if there’s at least one stage that passes (helps
when reducing the error for bug report purposes).

When the Release Manager sends you the release candidate, download all sources,
unzip on the same directory (there will be sym-links from the appropriate places
to them), and run the release test as above.

If you found regressions or failures when comparing a release candidate with the
previous release, follow the rules below:

Critical bugs on compilation should be fixed as soon as possible, possibly
before releasing the binary blobs.

Check-all tests should be fixed before the next release candidate, but can
wait until the test-suite run is finished.

Bugs in the test suite or unimportant check-all tests can be fixed in between
release candidates.

New features or recent big changes, when close to the release, should have
done in a way that it’s easy to disable. If they misbehave, prefer disabling
them than releasing an unstable (but untested) binary package.