Reproducing Go binaries byte-by-byte

Fully reproducible builds are important because they bridge the gap between auditable open source and convenient binary artifacts. Technologies like TUF and Binary Transparency provide accountability for what binaries are shipped to users, but that's of limited utility if there is no way (short of reverse engineering) of proving that the binary is in fact the result of compiling the intended source.

That's why the Debian project is putting tremendous effort into making packages reproducible. The good news is that Go builds are reproducible by default.

Prerequisites

There are a few common sense requirements.

Of course, the builds must be reproducible in the weaker sense: that means the source code must match perfectly.

This includes dependencies, so the project has to vendor them strictly. This is important beyond binary reproducibility: you don't want for "version 1.3" of a software to mean different things based on when it was built.

The compiler version must be the same.

GOPATH and GOROOT must match (#16860), annoyingly, as they are all over the binary in debug file paths.

Note: the default GOROOT, the one that the compiler will use if the environment variable is not set, must also match, since it will be copied into binaries (#17943). You can only change that by recompiling the toolchain in the right directory.

Detecting parameters

To start, we need to figure out the GOPATH and GOROOT values they were built with. This is easy to figure out using debug/gosym and debug information to query the file path of known functions. (PE support is... left as an exercise to the reader.)

Debugging

The first thing to look at is the Build ID. The Build ID is a hash of the filenames of the compiled files, plus the version of the compiler (and other things in zversion.go, like the default GOROOT). See pkg.go.

What got me with rclone was not rebuilding the compiler in the new location to get the right default GOROOT—the make.bash step of the Dockerfile. If you enjoy debugging, here's the tootstorm on Mastodon.