A little over six months ago, I wrote about how to
enable sane distribution of CFFI. That previous post contained a number of
work arounds and hacks to deal with a single design decision in CFFI, namely
that it would implicitly invoke a compiler to compile your module and that was
a core part of the API. A little over a month ago, CFFI 1.0 was released which
offered new APIs which changed that assumption and offered better integration
with setuptools. There are still a few things to keep in mind while writing
a CFFI using module to enable easy and sane distribution, however it is now
much easier to do so.

I’m going to adapt the original examples from my previous post to use the new
APIs so we can see how it’s changed, and what the new best way of distributing
a CFFI based project is.

Minimal Example

Here is a minimal example of using CFFI to be able to call the printf
function from Python:

# This file should be saved as example_build.py
from cffi import FFI
ffi = FFI()
ffi.cdef(
"""
int printf(const char *format, ...);
"""
)
ffi.set_source(
"_example", # This is the name of the import that this will build.
"""
#include <stdio.h>
"""
)
if __name__ == '__main__':
ffi.compile()

# This file should be saved as example.py
from _example import ffi, lib
if __name__ == "__main__":
lib.printf(b"Hi There!\n")

This example works, and if you save both files into your current directory you
can verify it by running:

$ python example_build.py
$ python example.py
Hi There!

This works because when you first execute example_build.py it will
construct a FFI object at the module scope, and then execute the compile()
method on that FFI object. This will cause CFFI to compile the _example.so
module which is a standard Python extension module that you can simply import.
This can let you quickly and easily write simple modules with a minimal amount
of overhead.

Packaging our Example Project

Now that we have a simple example.py file we can package this up so that we
can distribute it to other people. We’ll use a simple setup.py taken from
the CFFI docs with some slight modifications to fit our project:

Now that we have our setup.py we can go ahead and create a sdist using the
command python setup.py sdist which will give us example-0.1.tar.gz in
the dist/ folder. We can even publish it to PyPI and then let other users
install it using pip install example!

Right about here is where my previous post started to layer more and more hacks
ontop of everything in order to restore some sanity to distributing. The good
news is CFFI 1.0 fixed all of this and we’re already done! People installing
this distribution will require a few system dependencies like the Python
development headers and libffi and its development headers however there is no
longer a need for all of the layers of monkeypatching and hacks.

The one really subtle thing I would point out here that isn’t obvious in our
example, is that you should not install the build scripts. When you’re simply
shipping a single .py file (such as in the example) then you can handle
this by simply not adding the example_build.py (or whatever name your
script has) to the py_modules list. However if you’re instead packaging
an importable package (e.g. modules inside of a directory) then you would
instead want do something like this:

# This should be saved as _cffi_build/example_build.py
from cffi import FFI
ffi = FFI()
ffi.cdef(
"""
int printf(const char *format, ...);
"""
)
ffi.set_source(
"_example", # This is the name of the import that this will build.
"""
#include <stdio.h>
"""
)

# This should be saved as example/__init__.py
from example._example import ffi, lib
if __name__ == "__main__":
lib.printf(b"Hi There!\n")

This will have the same outcome as the first example, you’ll get the example
project installed without installing the build script.

Bonus: “Better” setup_requires

Sadly, a better CFFI still doesn’t solve the issues around setup.py and
setuptools, particularly that the setup.py as written above will install
CFFI and all of its dependencies for any invocation of setup.py, even just
for printing out the usage information with python setup.py --help. The
setup_requires dependencies exist there to allow CFFI to introduce the
cffi_modules keyword, however setuptools doesn’t know in which cases you
actually want to install the setup_requires and in which cases they are
superflous, so it just always installs them.

We can limit this so that setuptools will only install CFFI if required,
however it requires adding more logic to our setup.py. This isn’t strictly
required though users may appreciate being able to query information from the
setup.py without downloading and installing CFFI.

To do this we’ll create a function that will inspect the arguments that
setup.py was called with and determine if any of them are invoking
something which will require CFFI in setup_requires. This function can then
add additional keyword arguments to the setup() function call depending on
if we need CFFI in the setup_requires or not.