Building a Pixel kernel with KASAN+KCOV

Kernel Address Sanitizer (KASAN)
helps kernel developers and testers find runtime memory-related bugs, such as
out-of-bound read or write operations, and use-after-free issues. While KASAN
isn't enabled on production builds due to its runtime performance
penalties and memory usage increment, it is still a valuable tool for testing
debug builds.

When used with another runtime tool called Kernel Coverage (KCOV), KASAN-sanitized and
KCOV-instrumented code helps developers and testers to detect runtime memory
errors and obtain code coverage information. In the scenario of kernel fuzz
testing, e.g. via syzkaller,
KASAN helps to determine the root cause of crashes, and KCOV provides code
coverage information to the fuzzing engine to help in test-case or corpus
deduplication.

This page does not discuss the inner workings or mechanics of KASAN. Rather, it
serves as a guide to build and modify the Android Open Soure Project (AOSP) and
Pixel's kernel source to boot with KASAN and KCOV turned on.

Setting up your build
environment

Building AOSP

Download the Android source code. For the
purpose of building KASAN images, choose a stable build that is not in active
development. Often, the latest available release/stable branch is a good choice.
More information about build and branch can be found at Source
Code Tags and Builds.

After you have successfully checked out the source code, download the necessary
device blobs that correspond to the device and branch you are using from Driver Binaries for
Nexus and Pixel Devices. Download both the vendor image and the set of
binaries from the System-on-Chip (SOC) manufacturer. Then, unarchive the
downloaded tarballs, run the scripts they contain, and accept the licenses.

Tip: Double check that you have the right
version of JDK installed on your system before proceeding further.

Then clean up, set up your build environment, and choose your build target,
following the steps in
Preparing to Build.

To establish a working base, make your first build without modifications:

make -j48

Flash your build result to a test device (for example, marlin) and let it boot:

There's an internal problem with your device. Contact your manufacturer
for details. This pop-up likely means that the build fingerprint of your
vendor and your system partition do not match. Because this build is just for
development and testing, and not a release build, just ignore it.

Building the kernel

To build the kernel, you need to check out the correct source code,
cross-compile it, and then build the kernel image in the correct AOSP
directory.

Checking out kernel source code

Create a directory to store the kernel source code and clone the AOSP kernel git
repository to your local storage.

Enter the msm directory and git checkout the branch
that corresponds to the source code you are building. For the list of available
branches and tags, see the Android msm
kernel source tree.

cd msmgit checkout TAG_NAME

After completing this step, the msm directory should have content.

Performing cross compilation

Next you need to compile the Android kernel.

Setting up your cross-compiler

To build the kernel, you need to set up a cross-compiler. The current
recommended and tested toolchain is Android's NDK toolchain latest stable
version. To download the Android NDK, visit the official Android NDK
website. Download the appropriate zip archive for your platform, and unzip
it. This results in a directory resembling
android-ndk-NDK_VERSION.

Downloading the LZ4c tool

The Pixel kernel uses LZ4 compression,
so the lz4c tool is required when you build your kernel. If you
use Ubuntu, install the lz4c tool by:

sudo apt-get install liblz4-tool

Building your kernel

Set up your build environment from the marlin-kernel-src/msm
directory with:

After flashing, your device should boot. Verify the image you flashed to
the device is the kernel image you built by checking Kernel
version under Settings -> System -> About phone
after the device finishes booting.

Modifying the kernel

Enabling KASAN and KCOV compile
options

KASAN and KCOV codes are guarded by compilation flags, which are not turned on
for normal builds. To enable them, add KASAN and KCOV options to the config
file, but drop the LZ4 config.

To do this, make a copy of the default config file, for example,
marlin_defconfig:

cd arch/arm64/configscp marlin_defconfig marlin-kasan_defconfig

In the new config file, remove this flag CONFIG_KERNEL_LZ4=y and
add these flags:

To see if KCOV was properly compiled, perform additional analysis on the
produced vmlinux at the root of the kernel source tree. If you run
an objdump on vmlinux, you should see numerous calls to
__sanitizer_cov_trace_pc().

Modifying AOSP code

Before plugging in the new boot image, you need to adjust certain parameters in
AOSP's source code that govern how the device boots. This is mainly necessary to
ensure the new (inflated) image boots properly.

Adjusting board parameters

Adjust the boot parameters defined in the device's BoardConfig.mk
file. It is located at device/google/marlin/marlin relative to the
root of your AOSP source code.

cd device/google/marlin/marlinvim BoardConfig.mk

Caution: Make sure you have a backup of the original
BoardConfig.mk file before proceeding, in case something goes
wrong.

The adjustments to be made can be summarized as follows through a
git diff result:

If you do not wish to modify the BoardConfig.mk
file, you could instead create a new lunch target that contains the name
marlin_kasan. For more information about this process, see
Adding a New Device.

Adjusting the kernel
target in the local Makefile

The new kernel uses LZ4 compression to improve speed, but KASAN requires gzip
for better compression ratio. To accommodate this, tell the build machinery
which kernel to bundle into the final target by modifying where the
LOCAL_KERNEL variable points to in
device/google/marlin/device-common.mk.

Rebuilding the boot image

To rebuild the boot image, copy the new kernel image into the AOSP tree in the
device-specific folder (e.g. device/google/marlin-kernel). Make
sure this is where the build system expects the kernel target image to be
at, according to how you modified it earlier.

Next, rebuild your flashable images, similar to how you built AOSP earlier. Upon successful build, flash all
built images as usual.

Booting your device
with a modified kernel image

You should now have a build that boots and enters the home screen. From here,
check the device's dmesg output for a "KernelAddressSanitizer
initialized" message in the very early boot stage. That message means
KASAN is initialized during boot time. Also, you can confirm
/sys/kernel/debug/kcov is present on the device (you will have to
be root to do that).

Troubleshooting

You can experiment with different kernel versions, starting with a standard
build as a working base, before turning on KASAN+KCOV compile options. When
things break, first check if the bootloader and baseband version on your device
matches those required by the new build. Finally, you might have to
catch up with a newer branch of the Android tree altogether if you venture too
far ahead with the kernel version.