Enabling `perf` in Kubernetes with Docker’s default seccomp profile

December 21, 2018

Enabling `perf` in Kubernetes with Docker’s default seccomp profile

Have you been trying to profile your Kubernetes applications with perf? Maybe you want to see what all the FlameGraphs fuss is about? If your version of Docker was upgraded within the last year, you’ll likely run into issues.

Trying to alter the suggested /proc/sys/kernel/perf_event_paranoid from within the container gets you the expected:

bash: /proc/sys/kernel/perf_event_paranoid: Read-only file system

What to do? You’ll need to enable CAP_SYS_ADMIN. This flag is one of many Linux capabilities, so named for the extra capabilities they grant. These flags grant scoped permission escalations for threads to perform specific tasks, from changing file attributes to altering the system clock. CAP_SYS_ADMIN is a particularly overloaded one, a kitchen sink of permissions escalations mostly geared toward profiling work.

If you’re only working with Docker, you can add --cap-add SYS_ADMIN to your docker run command, as explored here.

However, if you’re living that Kubernetes life, you’ll need to enable it using a securityContext. In the container spec of your deployment file, add:

Remember to remove this setting when you’re done using it!perf_event_open is blocked by default because it grants user processes privileged access to the system. Branch deploy your change, use it, then rollback.

Alice Goldfuss

Alice Goldfuss is a systems punk currently helping GitHub run their cutting-edge container platform. She loves kernel crashes, memory design, and performance hacks.

Alice has consulted on some books (Docker: Up & Running, Effective DevOps, Site Reliability Engineering vol 2), presented at some conferences (SREcon, Velocity, Container Summit), and run some others (LISA17, DevOps Days Portland). You can follow her on Twitter (@alicegoldfuss), but you’ll probably regret it.