Sanitizing C++ Python Modules

Jonas Devlieghere

Python has great interoperability with C and C++ through extension modules. There are many reasons to do this, such as improving performance, accessing APIs not exposed by the language, or interfacing with libraries written in C or C++.

Unlike Python however, C and C++ are not memory safe. Luckily, great tools exist to help diagnose these kind of issues. One of those tools is ASan (Address Sanitizer) which uses compiler instrumentation to detect memory errors at runtime.

Interceptors

There are some things to be aware of when using a ASanified module from Python. For the sake of this post I'm going to assume you're running macOS Catalina and are using Python 3 that comes with Xcode (/usr/bin/python3 or xcrun python3).

When you first import the module, you might encounter the following error:

$ /usr/bin/python3
>>> import sanitized
===12345===ERROR: Interceptors are not working. This may be because AddressSanitizer is loaded too late (e.g. via dlopen). Please launch the executable with:
DYLD_INSERT_LIBRARIES=/path/to/libclang_rt.asan_osx_dynamic.dylib

For ASan to work, it needs to intercept functions like malloc and free to track memory usage. This requires the runtime to be loaded first, before the library (e.g. libc) that exports these functions.

When importing a sanitized module in Python, the dynamic linker (dyld) will have already loaded these symbols, before it dynamically loads the sanitized module with dlopen.

In the error message, it's nice enough to suggest using the environment variable DYLD_INSERT_LIBRARIES to change the dynamic linker's behavior. This allows dyld to load libraries, specifically the sanitizer runtime, before loading the current binary.

System Integrity Protection

$ DYLD_INSERT_LIBRARIES=/path/to/libclang_rt.asan_osx_dynamic.dylib /usr/bin/python3
>>> import sanitized
===12345===ERROR: Interceptors are not working. This may be because AddressSanitizer is loaded too late (e.g. via dlopen). Please launch the executable with:
DYLD_INSERT_LIBRARIES=/path/to/libclang_rt.asan_osx_dynamic.dylib

If your first instinct is checking that the environment variable actually makes it to Python you wouldn't be alone. If you have System Integrity Protection (SIP) enabled, you're in for some extra confusion. According to Apple's documentation:

Spawning children processes of processes restricted by System Integrity Protection [...] resets the Mach special ports of that child process. Any dynamic linker (dyld) environment variables, such as DYLD_LIBRARY_PATH, are purged when launching protected processes.

We can use another environment variable to verify that the sanitizer runtime is indeed loaded first. With DYLD_PRINT_LIBRARIES, dyld will print all the libraries as they are loaded.

Shim

The problem is a little less obvious this time. Remember how I assumed we're using Python 3 from Xcode? You might wonder why we're using /usr/bin/python3 and not something living in /Applications/Xcode.

If we run nm on the binary we can see that it's actually a small shim. Without having Xcode installed, it'll instruct you to download and install it. Otherwise, it simply forwards to the Python binary in Xcode.

What's happening here is that we launch the shim with the ASan runtime loaded first, but then launch the real interpreter and because the environment variable doesn't get forwarded, the runtime is only loaded when we import the sanitized module, which is too late.

Yup, another shim. This time it's using posix_spawn, but the problem is exactly the same as before. Let's use lldb to find out what binary is spawned. We know that the second argument contains the path to the binary to launch.

Finally, when we launch the path pointing into the Python 3 Framework, everything works as expected.

Conclusion

As you can imagine, it was a lot of fun figuring this out. The most ironic part was that we had a similar issue with Python 2 and worked around it, but forgot the underlying issue. When we moved our CI to Python 3 we got to enjoy this treasure hunt all over again! A wholehearted thank you to my colleagues Adrian Prantl and Dan Liew for helping me figure this one out.

The workaround for Python 2 consisted of re-launching Python with the path returned by sys.executable. From Python 2, this would return the real interpreter binary. From Python 3 this returns the second shim.

Vedant Kumar came up with a nifty Python script to peel away the Python shims and reveal the actual interpreter binary on macOS: