Sheriff brief guideline

The main idea of the bots is to use Clang as the compiler and its instrumentation, Address Sanitizer.

AddressSanitizer (ASAN) is a fast memory error detector based on compiler instrumentation (LLVM). It is fully usable for Chrome on Linux and macOS.

If the bot goes red on VMTest step or Archive step, look up artifacts. They should contain top-level files asan_log.1234.txt where 1234 is the PID of the failing process. Logs contain detailed information about the failure. Use it to file a bug, see this one as an example.

Note that Archive step will go red if there is at least one asan_log file found. This is made so because I noticed a rare case when ASAN caught a bug (and created log), but no test failed (and VMTest step remained green).

How to suppress a bug

When you filed a bug, but don't want to revert a change for some reason, you may suppress the offending function (top of your stack trace) like this:

// Mask it out for ASAN before the bug is fixed, see crbug.com/...

#if defined(ADDRESS_SANITIZER)

__attribute__((no_address_safety_analysis))

#endif // defined(ADDRESS_SANITIZER)

void BuggyFunction() { ... }

Note that, as a result, the bug may become a SEGV and the bot will still be red. If this happens and you want to suppress it, you may add some filtering to the buildbot code that make bot red after asan_log: see GenerateStackTraces() in chromite/cbuildbot/commands.py.

Details

All mentioned Chromium OS ASAN bots are doing the same thing with slight differences:

Note that Chromium OS waterfall bots (1-2) build last known good Chrome, so if the ASAN bug creeps into Chrome code, bots 3-4 will go red immediately, while bots 1-2 may still be green for a day or so.

Following are the differences between Chromium OS ASAN bots and similar incremental / pgq bots (like amd64-generic-incremental / amd64-generic-tot-chrome-pfq-informational).

BuildTarget step differences

The bots use asan profile (chromeos/src/overlays/overlay-amd64-generic/profiles/asan/) to override some settings.
Mostly it adds USE="asan". So only packages that check that flag are built with Clang/ASAN.

If you ever need to try asan profile locally, do

setup_board --board=amd64-generic --profile=asan --force

then run build_packages as usual. Note that most boards do not have an asan profile. For other boards you may do manually what asan profile says.

Archive step differences

As mentioned above, Archive step goes red if there is at least one asan_log file in /var/log/chrome directory. Additionally, for every such file, the log is symbolized and put to the top-level artifacts. Note that logging_AsanCrash test remove its asan_logs so they don't break things.

Run time differences

Session manager adds some environment variables to the corresponding processes environment. See "SetUpASAN" and below.

The most important is that it restricts Chrome from using sandbox (adds --no-sandbox flag).

Updating Clang revision periodically

We keep Chromium OS clang and its dependent llvm packages updated. As a reference, I use Gentoo repo of sys-devel. When updating local clang, I try to make it as close as possible to Gentoo.

As a signal to update Chromium OS Clang I watch at Chrome's tools/clang/scripts/update.sh (added myself to the WATCHLIST for 'clang_update'). Chrome Clang guys do the good job to test the new version before they commit.

Currently Chromium OS ASAN bots are the only users of Clang in Chromium OS. However, llvm package is also used by mesa.

From the bots point of view, we want to use the most recent stable version of Clang to have the most edge ASAN features working. From the mesa point of view, edge llvm may not work. So we use chromiumos-overlay/profiles/targets/chromeos/package.mask to restrict latest llvm to be pulled to target chroots. Remember to update it when llvm version changes.

SVN <-> Git mapping

The git clang/llvm repos are pulled from git. Since the main llvm repo is maintained in subversion, and the the git repos are mirrors of the svn repo, and the ebuilds are versioned based on the svn repo, you have to map the two.

Every git commit has metadata in it that tells you the svn rev it maps to. As an example:

With this information in hand, you can update the git commit sha1 in the ebuilds.

Note that since upstream llvm has a single svn repo for all of its projects, the svn rev in the git tree might not be there exactly. So you have to find the commit in there that is closet to (but does not exceed) the svn rev you want.