Thursday, May 21, 2015

CFFI 1.0.1 final has now been released for CPython! CFFI is a (CPython and PyPy) module to interact with C code from Python.

The main news from CFFI 0.9 is the new way to build extension modules:
the "out-of-line" mode, where you have a separate build script. When
this script is executed, it produces the extension module. This comes
with associated Setuptools support that fixes the headache of
distributing your own CFFI-using packages. It also massively cuts
down the import times.

Although this is a major new version, it should be fully
backward-compatible: existing projects should continue to work, in
what is now called the "in-line mode".

The documentation has been reorganized and split into a few pages.
For more information about this new "out-of-line" mode, as well as
more general information about what CFFI is and how to use it, read the Goals and proceed to
the Overview.

PyPy support: PyPy needs integrated support for efficient JITting,
so you cannot install a different version of CFFI on top of an
existing PyPy. You need to wait for the upcoming PyPy 2.6 to use
CFFI 1.0---or get a nightly build.

My thanks again to the PSF (Python Software Foundation) for their
financial support!

CFFI 1.0.1 final has now been released for CPython! CFFI is a (CPython and PyPy) module to interact with C code from Python.

The main news from CFFI 0.9 is the new way to build extension modules:
the "out-of-line" mode, where you have a separate build script. When
this script is executed, it produces the extension module. This comes
with associated Setuptools support that fixes the headache of
distributing your own CFFI-using packages. It also massively cuts
down the import times.

Although this is a major new version, it should be fully
backward-compatible: existing projects should continue to work, in
what is now called the "in-line mode".

The documentation has been reorganized and split into a few pages.
For more information about this new "out-of-line" mode, as well as
more general information about what CFFI is and how to use it, read the Goals and proceed to
the Overview.

PyPy support: PyPy needs integrated support for efficient JITting,
so you cannot install a different version of CFFI on top of an
existing PyPy. You need to wait for the upcoming PyPy 2.6 to use
CFFI 1.0---or get a nightly build.

My thanks again to the PSF (Python Software Foundation) for their
financial support!

Tuesday, May 5, 2015

Finally! CFFI 1.0 is almost ready. CFFI gives Python developers a convenient way to call external C libraries. Here "Python" == "CPython or PyPy", but this post is mostly about the CPython side of CFFI, as the PyPy version is not ready yet.

On CPython, you can download the version
"1.0.0b1" either by looking for the cffi-1.0 branch in
the repository, or by
saying

pip install "cffi>=1.0.dev0&quot

(Until 1.0 final is ready,
pip install cffi will still give you version 0.9.2.)

The main news: you can now explicitly generate and compile a CPython C
extension module from a "build" script. Then in the rest of your
program or library, you no longer need to import cffi at all.
Instead, you simply say:

from _my_custom_module import ffi, lib

Then you use ffi and lib just like you did in your
verify()-based project in CFFI 0.9.2. (The lib is what used to
be the result of verify().) The details of how you use them
should not have changed at all, so that the rest of your program should
not need any update.

Benefits

This is a big step towards standard practices for making and
distributing Python packages with C extension modules:

on the one hand, you need an explicit compilation step, triggered
here by running the "build" script;

on the other hand, what you gain in return is better control over
when and why the C compilation occurs, and more standard ways to write
distutils- or setuptools-based setup.py files (see below).

Additionally, this completely removes one of the main drawbacks of using
CFFI to interface with large C APIs: the start-up time. In some cases
it could be extreme on slow machines (cases of 10-20 seconds on ARM
boards occur commonly). Now, the import above is instantaneous.

In fact, none of the pure Python cffi package is needed any more at
runtime (it needs only an internal extension module from CFFI, which
can be installed by doing "pip install cffi-runtime" [*] if you only need that).
The ffi object you get by the import above is of a
completely different class written entirely in C. The two
implementations might get merged in the future; for now they are
independent, but give two compatible APIs. The differences are that
some methods like cdef() and verify() and set_source() are
omitted from the C version, because it is supposed to be a complete FFI
already; and other methods like new(), which take as parameter a
string describing a C type, are faster now because that string is parsed
using a custom small-subset-of-C parser, written in C too.

In practice

CFFI 1.0 beta 1 was tested on CPython 2.7 and 3.3/3.4, on Linux and to
some extent on Windows and OS/X. Its PyPy version is not ready yet,
and the only docs available so far are those below.

This is beta software, so there might be bugs and details may change. We are interested in hearing any feedback (irc.freenode.net #pypy) or bug reports.

To use the new features, create a source file that is not imported by the rest of
your project, in which you place (or move) the code to build the FFI
object:

The ffi.set_source() replaces the ffi.verify() of CFFI 0.9.2.
Calling it attaches the given source code to the ffi object, but this call doesn't
compile or return anything by itself. It may be placed above the ffi.cdef()
if you prefer. Its first argument is the name of the C extension module
that will be produced.

Actual compilation (including generating the complete C sources) occurs
later, in one of two places: either in ffi.compile(), shown above,
or indirectly from the setup.py, shown next.

If you directly execute the file foo_build.py above, it will
generate a local file _foo.c and compile it to _foo.so (or the
appropriate extension, like _foo.pyd on Windows). This is the
extension module that can be used in the rest of your program by saying
"from _foo import ffi, lib".

Distutils

If you want to distribute your program, you write a setup.py using
either distutils or setuptools. Using setuptools is generally
recommended nowdays, but using distutils is possible too. We show it
first:

This is similar to the CFFI 0.9.2 way. It only works if cffi was
installed previously, because otherwise foo_build cannot be
imported. The difference is that you use ffi.distutils_extension()
instead of ffi.verifier.get_extension(), because there is no longer
any verifier object if you use set_source().

Setuptools

The modern way is to write setup.py files based on setuptools, which
can (among lots of other things) handle dependencies. It is what you
normally get with pip install, too. Here is how you'd write it:

the first time is in setup_requires, which means that cffi will
be locally downloaded and used for the setup.

the second mention is a custom cffi_modules argument. This
argument is handled by cffi as soon as it is locally downloaded. It
should be a list of "module:ffi" strings, where the ffi part
is the name of the global variable in that module.

the third mention is in install_requires. It means that in
order to install this example package, "cffi-runtime" must also be
installed. This is (or will be) a PyPI entry that only contains a
trimmed down version of CFFI, one that does not include the pure
Python "cffi" package and its dependencies. None of it is needed at
runtime.

[*] NOTE: The "cffi-runtime" PyPI entry is not ready yet. For now, use "cffi>=1.0.dev0" instead. Considering PyPy, which has got a built-in "_cffi_backend" module, the "cffi-runtime" package could never be upgraded there; but it would still be nice if we were able to upgrade the "cffi" pure Python package on PyPy. This might require some extra care in writing the interaction code. We need to sort it out now...

Thanks

Special thanks go to the PSF (Python Software Foundation) for their
financial support, without which this work---er... it might likely have occurred anyway, but at an unknown future date :-)

(For reference, the amount I asked for (and got) is equal to one
month of what a Google Summer of Code student gets, for work that will
take a bit longer than one month. At least I personally am running mostly
on such money, and so I want to thank the PSF again for their
contribution to CFFI---and while I'm at it, thanks to all other
contributors to PyPy---for making this job more than an unpaid hobby on
the side :-)

Armin Rigo

Finally! CFFI 1.0 is almost ready. CFFI gives Python developers a convenient way to call external C libraries. Here "Python" == "CPython or PyPy", but this post is mostly about the CPython side of CFFI, as the PyPy version is not ready yet.

On CPython, you can download the version
"1.0.0b1" either by looking for the cffi-1.0 branch in
the repository, or by
saying

pip install "cffi>=1.0.dev0&quot

(Until 1.0 final is ready,
pip install cffi will still give you version 0.9.2.)

The main news: you can now explicitly generate and compile a CPython C
extension module from a "build" script. Then in the rest of your
program or library, you no longer need to import cffi at all.
Instead, you simply say:

from _my_custom_module import ffi, lib

Then you use ffi and lib just like you did in your
verify()-based project in CFFI 0.9.2. (The lib is what used to
be the result of verify().) The details of how you use them
should not have changed at all, so that the rest of your program should
not need any update.

Benefits

This is a big step towards standard practices for making and
distributing Python packages with C extension modules:

on the one hand, you need an explicit compilation step, triggered
here by running the "build" script;

on the other hand, what you gain in return is better control over
when and why the C compilation occurs, and more standard ways to write
distutils- or setuptools-based setup.py files (see below).

Additionally, this completely removes one of the main drawbacks of using
CFFI to interface with large C APIs: the start-up time. In some cases
it could be extreme on slow machines (cases of 10-20 seconds on ARM
boards occur commonly). Now, the import above is instantaneous.

In fact, none of the pure Python cffi package is needed any more at
runtime (it needs only an internal extension module from CFFI, which
can be installed by doing "pip install cffi-runtime" [*] if you only need that).
The ffi object you get by the import above is of a
completely different class written entirely in C. The two
implementations might get merged in the future; for now they are
independent, but give two compatible APIs. The differences are that
some methods like cdef() and verify() and set_source() are
omitted from the C version, because it is supposed to be a complete FFI
already; and other methods like new(), which take as parameter a
string describing a C type, are faster now because that string is parsed
using a custom small-subset-of-C parser, written in C too.

In practice

CFFI 1.0 beta 1 was tested on CPython 2.7 and 3.3/3.4, on Linux and to
some extent on Windows and OS/X. Its PyPy version is not ready yet,
and the only docs available so far are those below.

This is beta software, so there might be bugs and details may change. We are interested in hearing any feedback (irc.freenode.net #pypy) or bug reports.

To use the new features, create a source file that is not imported by the rest of
your project, in which you place (or move) the code to build the FFI
object:

The ffi.set_source() replaces the ffi.verify() of CFFI 0.9.2.
Calling it attaches the given source code to the ffi object, but this call doesn't
compile or return anything by itself. It may be placed above the ffi.cdef()
if you prefer. Its first argument is the name of the C extension module
that will be produced.

Actual compilation (including generating the complete C sources) occurs
later, in one of two places: either in ffi.compile(), shown above,
or indirectly from the setup.py, shown next.

If you directly execute the file foo_build.py above, it will
generate a local file _foo.c and compile it to _foo.so (or the
appropriate extension, like _foo.pyd on Windows). This is the
extension module that can be used in the rest of your program by saying
"from _foo import ffi, lib".

Distutils

If you want to distribute your program, you write a setup.py using
either distutils or setuptools. Using setuptools is generally
recommended nowdays, but using distutils is possible too. We show it
first:

This is similar to the CFFI 0.9.2 way. It only works if cffi was
installed previously, because otherwise foo_build cannot be
imported. The difference is that you use ffi.distutils_extension()
instead of ffi.verifier.get_extension(), because there is no longer
any verifier object if you use set_source().

Setuptools

The modern way is to write setup.py files based on setuptools, which
can (among lots of other things) handle dependencies. It is what you
normally get with pip install, too. Here is how you'd write it:

the first time is in setup_requires, which means that cffi will
be locally downloaded and used for the setup.

the second mention is a custom cffi_modules argument. This
argument is handled by cffi as soon as it is locally downloaded. It
should be a list of "module:ffi" strings, where the ffi part
is the name of the global variable in that module.

the third mention is in install_requires. It means that in
order to install this example package, "cffi-runtime" must also be
installed. This is (or will be) a PyPI entry that only contains a
trimmed down version of CFFI, one that does not include the pure
Python "cffi" package and its dependencies. None of it is needed at
runtime.

[*] NOTE: The "cffi-runtime" PyPI entry is not ready yet. For now, use "cffi>=1.0.dev0" instead. Considering PyPy, which has got a built-in "_cffi_backend" module, the "cffi-runtime" package could never be upgraded there; but it would still be nice if we were able to upgrade the "cffi" pure Python package on PyPy. This might require some extra care in writing the interaction code. We need to sort it out now...

Thanks

Special thanks go to the PSF (Python Software Foundation) for their
financial support, without which this work---er... it might likely have occurred anyway, but at an unknown future date :-)

(For reference, the amount I asked for (and got) is equal to one
month of what a Google Summer of Code student gets, for work that will
take a bit longer than one month. At least I personally am running mostly
on such money, and so I want to thank the PSF again for their
contribution to CFFI---and while I'm at it, thanks to all other
contributors to PyPy---for making this job more than an unpaid hobby on
the side :-)