Creating Python-related Ebuilds

Summary

Funtoo Linux incorporates Progress Overlay into the main Portage tree, which currently contains 400+ updated Python-related packages that implement next-generation Python support for ebuilds.

Building Python Modules with Portage

Conceptual Overview

Portage supports the ability to build python modules and extensions for all installed versions of python. This functionality has been implemented by python.eclass.

To use python.eclass, you first want to investigate how python module building is integrated into any included Makefile or build script. Since we want python.eclass to build these modules for us, typically the first step is to tweak the Makefile or configure script to explicitly *not* build python modules. We will be handling that process using python.eclass.

Another option, for those Python packages with a setup.py in the root source directory, is to use the distutils eclass. In this case, you generally do not need custom src_compile or src_install stages, as the eclass takes care of everything for you. However, you will still need to set the proper environment variables described below, as they are needed by python.eclass which is used by distutils.eclass. You will also want to set dependencies in DEPEND and RDEPEND as described in this article.

Progress Improvements

This section describes the differences in approach between the old (classic) way of handling Python-related ebuilds, and the new way, which uses the updated eclass. Also note that the distutils eclass benefits from these updates as well, as it uses python.eclass behind-the-scenes.

The old way: python-updater

With the older python.eclass in Gentoo (and that used to be in Funtoo Linux,) python modules are built for all installed versions of Python. However, no usable accounting information was stored by Portage so that it could "know" what versions of python the module was built for. This created a problem when new versions of Python are installed, as the new module would not be available for the new python, and Portage would not be aware of what needed to be done to fix this, if anything. This resulted in the creation of a tool called python-updater to rebuild all necessary python-related ebuilds so that they reflected the currently-installed versions of Python.

Note that python-updater-0.10 and later are aware of the new Python eclass, and will not unnecessarily reinstall any packages which set PYTHON_MULTIPLE_ABIS and EAPI="4-python", as manual rebuilding is not necessary.

TODO: add python-updater example

The new way: new python eclass

The new python.eclass from Arfrever has a new API that can be used to write Python ebuilds. When this API is used, Portage can track and resolve missing builds of Python modules internally, so python-updater is not needed to be used for these ebuilds. However, until all ebuilds are converted to the new API, python-updater will need to be used to update these old-style ebuilds. Fortunately for us, over 400 Python-related ebuilds have been converted to take advantage of the new eclass, with more added weekly.

Setting up progress overlay

Note

Progess overlay is now merged in Funtoo current and stable - No manual steps are needed to benefit from it!

Notice, that progress overlay including an updated packages that also available in main portage tree!

# python-updater

this still required to rebuild old-fashion python-related ebuilds

Updating the packages from progress overlay

Package updates are done the same way than before:

emerge -avuDN @world

Several python-related packages should be updated to progress versions, with the new PYTHON_ABIS flags.

Developer Docs

Ebuild Converstion to new Python eclass

Conversion from previous EAPIs and eclasses to the new progress python eclass is fairly straightforward. This section tries to explain the steps.

4-python EAPI

First, set EAPI to 4-python:

EAPI=4-python

Next, you will need to understand the type of Python application or module you are trying to install. The primary determining factor in how to proceed is to see whether the Python application installs anything into Python's site-packages directory. Essentially all Python modules do this, and some applications do this as well. If a package installs stuff into site-packages, then you should have its ebuild take advantage of the Python eclass "multiple ABI" support.

In this context, "multiple ABI" refers to potentially multiple versions of Python that may be installed on your system. Each installed major.minor version of Python, plus Python derivatives like Jython and pypy, will have their own site-packages directory and the Python eclass can take care of installing a version of the Python modules for each version of Python on your system. If the application has binaries, a Python wrapper will also be installed that will hand off control to the proper version of Python automatically.

Non-Multiple-ABI Support

If you are not supporting multiple ABIs, which is the least common option, you will generally just set PYTHON_DEPEND to refer to the version(s) of Python you require. Documentation for PYTHON_DEPEND syntax (which has been changed in the new python eclass) is available here.

Multiple ABI Support

So, if you are using multiple ABI support, which will be the case for all Python modules, proceed as follows. Set PYTHON_MULTIPLE_ABIS prior to any inherit lines:

PYTHON_MULTIPLE_ABIS=1

Note the new variable names. PYTHON_MULTIPLE_ABIS turns on the functionality to build multiple versions of python modules for each version of python defined in the user's PYTHON_ABIS setting. Note that PYTHON_ABIS defaults to "2.7 3.2" in Funtoo Linux profiles and can be overridden in /etc/make.conf if you would like the python eclass to target other Python ABIs (like jython, for example, or 2.6) that you might be interested in.

When PYTHON_MULTIPLE_ABIS is set and EAPI is set to 4-python, then new USE flags will be automatically added to the IUSE with the python_abis_ prefix.
If you want to see what ABIs are supported, see the _PYTHON_GLOBALLY_SUPPORTED_ABIS variable defined in python.eclass.

Also note that for multiple ABI support, PYTHON_DEPEND is set to a reasonable default value so you should almost always never need to set it yourself.

PYTHON_RESTRICTED_ABIS

PYTHON_RESTRICTED_ABIS variable set in the ebuild specifies a space-separated list of patterns of Python ABIs not supported by the current package. The settings in PYTHON_RESTRICTED_ABIS will remove certain ABIs from the auto-IUSE expansion that happens when EAPI is set to 4-python and PYTHON_MULTIPLE_ABIS is set to 1 (This behavior is described above.) It will also limit the behavior of the python_abi_depend() function, described below. And of course, it will tell the python eclass to not build any python modules for these python versions, even if they happen to be installed on the system.

Use it like this, before the eclasses are inherited:

PYTHON_RESTRICTED_ABIS="3.*"

Note that PYTHON_RESTRICTED_ABIS is a glob-style pattern. If you want to have it build with everything except 3.2 and above and jython, use this pattern:

PYTHON_RESTRICTED_ABIS="3.[2-9] *-jython"

Note that the wildcard you should use to disable Jython is *-jython, while the wildcard you should use to disable PyPy is *-pypy-*.

python_abi_depend()

One of the cool things about the new Python eclass is the "cascading" dependency functionality which is implemented via USE variables. Basically, if your new-style python module ebuild was built for python-2.7, and it has dependencies, then it can depend on the module it needs being built for the installed version of python-2.7 specifically. This happens by having your ebuild depend upon the python_abis_2.7USE flag being set in the installed ebuild it depends upon.

For this functionality to work, the ebuild you depend on must be a new-style (EAPI 4-python) ebuild too, and must be enabled to support multiple python ABIS via the PYTHON_MULTIPLE_ABIS=1 setting. When these requirements are met, you will want to covert an old-style dependency that looks like this:

RDEPEND="foo >=dev-python/mydep-2 bar"

...to this new format:

RDEPEND="foo $(python_abi_depend ">=dev-python/mydep-2" ) bar"

The python_abi_depend function will expand to conditionally depend on the proper USE flags being set in dev-python/mydep-2 based on what USE flags are currently set for this ebuild when it builds. This is how the cascading functionality works. Use it in DEPEND, RDEPEND and PDEPEND as appropriate. You can include multiple ebuilds as arguments to python_abi_depend.

Please note that due to the fact that python_abi_depend is implemented as a shell function, the shell can be a bit picky about parsing its arguments. Follow these guidelines:

All arguments to python_abi_depend should be on the same line, or -

Additional lines should have a \ line continuation character as the last line, and -

Any atoms with > or < characters should be enclosed in double-quotes to avoid being interpreted as output/input redirection

python_abi_depend() gotchas

Occasionally, you may encounter emerge errors like this when developing new Python ebuilds:

xdev nova # emerge -p nova
These are the packages that would be merged, in order:
Calculating dependencies... done!
emerge: there are no ebuilds built with USE flags to satisfy "dev-python/lxml[python_abis_2.6?,python_abis_2.7?,python_abis_2.6-jython?,python_abis_2.7-pypy-1.7?,python_abis_2.7-pypy-1.8?]".
!!! One of the following packages is required to complete your request:
- dev-python/lxml-2.3.3::gentoo (Missing IUSE: python_abis_2.6-jython python_abis_2.7-pypy-1.7 python_abis_2.7-pypy-1.8)
(dependency required by "sys-cluster/nova-9999" [ebuild])
(dependency required by "nova" [argument])

Here's an explanation for why an error like this occurs. In this case, the lxml package has *-jython *-pypy-* in PYTHON_RESTRICTED_ABIS. Since nova depends on lxml, it should also be restricted so that it doesn't support Jython or PyPy. This makes sense, because if a dependency of nova can't run on a particular Python implementation, then neither can nova itself.

This emerge error can be fixed by adding *-jython *-pypy-* to nova's PYTHON_RESTRICTED_ABIS. The lesson here is that any Python-based application can only support the subset of all platforms supported by all its dependencies, and PYTHON_RESTRICTED_ABIS must reflect this.

On an unrelated note, nova is configured with PYTHON_MULTIPLE_ABIS=1 because it installs a ton of Python modules into site-packages.

common python_abi_depend() dependencies

This section is dedicated to providing examples on how to handle common dependencies that are found in Python-related packages:

argparse

If the Python package's setup.py contains something like this:

if sys.version_info < (2, 7):
requirements.append('argparse')

...then use the following dependency syntax:

(R)DEPEND="$(python_abi_depend virtual/python-argparse)"

python_abi_depend() with -i and -e

python_abi_depend() has some advanced capabilities that you may need to take advantage of related to Python dependencies. Sometimes, your ebuild will only need a python module if certain versions of Python are installed. This functionality is typically needed in the following cases:

module rolled into standard library

A module is needed, but in newer versions of python it is part of the standard library (so there is no need to depend on it for these newer python versions)

module required for fixes

A module is needed, but only for newer versions of python, such as 3.x, where it implements compatibility features.

optional module, not 100% compatible

A module can be used, but the dependencies are not supported with certain versions of Python.

Let's look at the first case, a module your ebuild needs that was rolled into the standard library starting with Python version 3.1. In this case, we only need to depend on this module for Python ABIs 3.0 or less. We would use the -e option to exclude the versions we are not interested in:

RDEPEND="$(python_abi_depend -e "3.[1-9]" dev-python/mydep)"

In the second case, we depend on a module only for Python ABIs 3 and up, since it implements fixes for python3. In this case, we will use the -i option to include only those versions for which we want to depend: