Playing with BPF

There’s a been a lot of partial resources and tips about using BPF, and it has already helped me to find an epic memory leak. I always wanted to get a chance to learn it deeper and the release of Gregg’s book was a good cause. My colleague from Shopify’s ProdEng team Dale Hamel helped to review the book too, which only pumped my interest about the technology.

When it comes to observability, the most impact (IMO) from being able to trace something really quickly and find the root cause is during incidents, when it’s not clear what’s happening and pulling the right observability tool from your hat is the top skill. That’s the muscle that I’d want to extract from the book, and writing notes in “what I’ve learned” format will hopefully help.

This post is a collection of notes taken while getting my hands dirty with each of the tools, as the book goes.

0. Getting bpfcc-tools ready

Turns out that apt-get install bpfcc-tools on Ubuntu 18.04 Bionic is broken, as I’m writing this in December 2019. Thankfully it’s fixed in the upstream, and I documented steps how to install a nightly package.

Turns out there’s a lot of stuff involved when you ssh and login into a terminal session.

2. biolatency

biolatency prints stats about I/O latency as a diagram. In the past, I’ve seen my colleagues using it to debug slow disks in cloud, which might be especially critical for latency-sensitive workloads like DBs.

Steps I’ve taken:

Install fio (I/O tester) on a VM with standard disk (pd-standard on Google Cloud) and run a sample test:

Under the hood, opensnoop is just a fancy wrapper around sudo bpftrace -e 'tracepoint:syscalls:sys_enter_open*.

However, tracing open() syscalls only shows files that were opened, but not files that were attempted to open but didn’t exist. This might come useful if you want to trace an app that’s failing due some config file that doesn’t exist.

A common scanerio could be:

ifFile.exist?(path/to/config)File.open(path/to/config)else# some other behaviourend

In my case, it was ProxySQL failing to start because it couldn’t find the config file:

Let’s find a syscall that attempts (and fails) to read /etc/proxysql.cnf. From what I found, it’s the access syscall that is used to check for file existance. It was easy enough to come up with a BPF instruction.

Yay! Now we can know that it’s the ProxySQL_ConfigFile8OpenFileEPK function that checks for existance of /etc/proxysql.cnf using access() syscall.

Inspecting failed DNS queries

I wanted to play more with the raw bpftrace -e and I came up with an actual problem from production that would be interesting to debug.
At work, I observed a tiny percentage of RPCs failing due DNS lookups timing out. I don’t have an idea yet why that might be happening, but with BPF I could at least observe those events better.

DNS resolution is done by getaddrinfo(3). I found gethostlatency.bt in bpftrace samples code which observes DNS resolution latency - something close to my case.

Now, with the probe running, we can try to hit curl google.com (which will fail on name lookup) and see the following output from bpftrace:

20:28:19 failed | 24087 curl (error -3) google.com

Hurray!

There’s probably another story for how to long-run that safely in production environment, which I’m yet to learn.

These are my notes for what I’ve learned so far from the BPF book, and I’m only 1/4 way through. Let’s see what the rest of the book brings.

The goal of publishing this is mostly for me to be able to come back and revisit some steps from what I’ve documented, but I hope it could be useful for others too.

I’m eager to learn if there’s anything here that I misunderstood or something that could be done better - please feel free to contact me on Twitter and point it out.

Also, thanks to my friend Javier Honduco for walking me over and explaining some of those things I’ve learned.

Tweet the link

About the author

Kir Shatrov helps businesses to grow by scaling the infrastructure. He likes to write about software, scalability and interesting stories that he runs into at work. Follow him on Twitter to get the latest updates: @kirshatrov.