Gentoo Python Developers Guide

Never bump one of the following packages yourself if permission to do so is not
explicitly granted by a (Co-)Lead:

dev-lang/python

app-admin/eselect-python

While doing version bumps is surely appreciated, don't do it blindly. There were
many bugs in the past which had been carried from version to version without
being noticed. Make also sure that you check bugzilla before the bump to
see whether there are open bugs for the package. Reading the ChangeLog of a
package is also a good idea to catch new, changed or optional dependencies.

Not all existing ebuilds in the tree use the eclasses properly (see below), so
please fix bugs on sight. Build the packages you're bumping or fixing even on
small changes. Not all ebuilds have been written carefully while others might
have been perfect when they have been committed. But over time, practice and
rules change.

The same goes for fixing bugs in the ebuilds. Please check whether there is a
new version of the package out and do a version bump accordingly. Closing bugs
is good, but not enough. Your primary objective should not be to close bugs but
to maintain the packages in the herd.

Ask for and do peer reviews. With such a practice, ebuild quality increases and
it is a good way to transfer knowledge.

Every team member should try to keep the package directories clean and uncluttered.
Besides the obvious checks (last stable for an arch, last not p.masked, other
packages depend on an exact version of this package), there are some other
things which you should consider before dropping an old version:

When dropping an unstable version in the presence of a stable one: Does the
version you are going to drop have serious bugs which avoid stabilization?
Otherwise you might keep it and open a stabilization bug.

The same consideration also applies if there is no stable version yet: Are
there users who might want a stable version? Is this package mature enough
to go stable? If you decide to stabilize it, also think about how arch team
members could test it.

Do not stabilize alpha and beta versions nor release candidates wherever
possible. There are exceptions to this (if upstream just produces beta-ware
or the package is desperately needed for another app). If unsure, talk to
the Lead first.

Packages supporting installation for multiple versions of Python install some files
(usually .py or .so) into directories specific to given
versions of Python.

Packages not supporting installation for multiple versions of Python usually
exhibit at least one of the following properties:

They install Python modules outside of site-packages directories.

They do not install any files into directories specific to given versions of Python.

They install libraries linked against Python library (e.g. libpython2.7.so)
outside of directories specific to given versions of Python (e.g. in /usr/lib)
and filenames of these libraries do not contain Python version.

They import Python modules installed by packages not supporting installation for
multiple versions of Python.

Ebuilds should properly specify dependency on supported version(s) of Python.
python.eclass supports PYTHON_DEPEND helper variable, which allows
to specify minimal and maximal version of Python. PYTHON_DEPEND
variable should be set before 'inherit'. PYTHON_DEPEND variable
should contain 1 or 2 groups of version components and can optionally begin with
USE flag conditional in the form of "flag? " or "!flag? ". Each group of version
components should contain major version ("2", "3" or "*") and can optionally contain
minimal version (e.g. "2.6") and maximal version (e.g. "3.1"). Version components
should be separated by colons. Colons followed only by empty components can be
ommitted. "*" major version means that versions of both Python 2 and Python 3 are
accepted. Minimal and maximal versions should contain major and minor versions.

Code Listing 2.1: Examples of PYTHON_DEPEND

# Dependency on any version of Python.
PYTHON_DEPEND="*"
# Dependency on any version of Python 2.
PYTHON_DEPEND="2"
# Dependency on any version of Python 3.
PYTHON_DEPEND="3"
# Dependency on any version of Python 2, which is at least 2.6.*.
PYTHON_DEPEND="2:2.6"
# Dependency on any version of Python 3, which is at least 3.2.*.
PYTHON_DEPEND="3:3.2"
# Dependency on any version of Python 2 or 3, which is at least 2.6.*.
PYTHON_DEPEND="*:2.6"
# Dependency on any version of Python 2, which is at least 2.7.*, or a version of Python 3, which is at least 3.2.*.
PYTHON_DEPEND="2:2.7 3:3.2"
# Dependency on any version of Python 2, which is at most 2.6.*.
PYTHON_DEPEND="2::2.6"
# Dependency on any version of Python 3, which is at most 3.2.*.
PYTHON_DEPEND="3::3.2"
# Dependency on any version of Python 2 or 3, which is at most 3.2.*.
PYTHON_DEPEND="*::3.2"
# Dependency on any version of Python 2, which is at least 2.5.* and at most 2.6.*.
PYTHON_DEPEND="2:2.5:2.6"
# Dependency on any version of Python 3, which is at least 3.1.* and at most 3.2.*.
PYTHON_DEPEND="3:3.1:3.2"
# Dependency on any version of Python 2 or 3, which is at least 2.6.* and at most 3.1.*.
PYTHON_DEPEND="*:2.6:3.1"
# Dependency on any version of Python 2, which is at least 2.5.* and at most 2.6.*, or a version of Python 3, which is at least 3.1.* and at most 3.2.*.
PYTHON_DEPEND="2:2.5:2.6 3:3.1:3.2"
# Dependency on any version of Python 2, when "python" USE flag is enabled.
PYTHON_DEPEND="python? 2"
# Dependency on any version of Python 2, which is at least 2.5.*, when "python" USE flag is enabled.
PYTHON_DEPEND="python? 2:2.5"
# Dependency on any version of Python 3, when "minimal" USE flag is disabled.
PYTHON_DEPEND="!minimal? 3"

Ebuilds can set PYTHON_USE_WITH or PYTHON_USE_WITH_OR
before 'inherit' and call python_pkg_setup() to check if Python has been
installed with specific USE flags. All USE flags specified in PYTHON_USE_WITH
must be enabled, but at least one USE flag specified in PYTHON_USE_WITH_OR must
be enabled. PYTHON_USE_WITH_OPT can specify a name of a USE flag,
which conditionalizes PYTHON_USE_WITH and PYTHON_USE_WITH_OR.
If python_set_active_version() (described below) is used, then it must be called
before python_pkg_setup().

Ebuilds not working with some versions of Python should set RESTRICT_PYTHON_ABIS
variable (e.g. after DEPEND/RDEPEND), which should contain list of space-separated
fnmatch patterns. Such patterns can contain '*' character.

Separate build directories must be used for different Python versions. Distutils
by default uses "build" directory, which can be changed by "-b" option of "build"
command of setup.py. Packages, which do not use Distutils, and very small number
of packages, which use Distutils, usually need to use build directories outside
of "${S}". Functions from distutils.eclass by default use "${S}/build-${PYTHON_ABI}"
build directories. Packages, which do not use "${S}/build-${PYTHON_ABI}" build
directories, need to call python_copy_sources() function, which copies
sources to separate build directories.

python_execute_function() is used to perform appropriate actions with all enabled
Python versions. This function requires one argument, which is name of function
or -d / --default-function option. This function accepts some optional arguments.
python_execute_function() executes a function, which needs to be defined earlier.
To improve readability, it's recommended to define functions, which are used only
in 1 place in ebuilds, directly before passing their names to python_execute_function().

-d / --default-function option is useful in cases, when the same
actions, which are executed in default phase functions (e.g. emake in src_compile()),
need to be executed. This option can be used only in a subset of ebuild phases.

Code Listing 2.9: Example of python_execute_function() with -d option

src_compile() {
python_execute_function -d -s
}

python.eclass defines the following phase functions, which can be used to simplify
some ebuilds:

python_src_prepare

python_src_configure

python_src_compile

python_src_test

python_src_install

python_src_prepare() calls 'python_copy_sources', while other phase functions call
'python_execute_function -d -s'. If PYTHON_EXPORT_PHASE_FUNCTIONS="1"
variable has been set before 'inherit', then these phase functions are exported.

PYTHON_ABI variable can be checked inside function executed by python_execute_function().

Important: --action-message and --failure-message options of python_execute_function()
accept arguments, which are internally evaluated, so single quotes might be useful.

Sometimes another eclass defines a specialized function, which performs building,
installation etc., but is designed for non-Python packages. In such cases, it's
possible to call python_execute_function() with name of such a function.

Code Listing 2.11: Example of python_execute_function() with a function from another eclass

If given package supports only Python 2 or only Python 3, then python_set_active_version()
function should be called to set active version of Python. Usually major version
of Python should be passed to this function.

Code Listing 2.12: Example of python_set_active_version() with major version of Python

pkg_setup() {
python_set_active_version 2
}

If given package supports only 1 version of Python, then Python ABI (in the form
of ${major_version}.${minor_version}) can be passed to python_set_active_version().
It will cause that python-updater will ignore this package.

Important:
To call Python interpreter in ebuilds, "$(PYTHON)" should be used.

In ebuilds supporting installation for multiple versions of Python, sometimes
given action needs to be executed only once (e.g. generation of documentation).
In such cases it should be executed with the final Python ABI from the list of
enabled ABI, which is performed by passing -f / --final-ABI option
to appropriate getter functions.

Shebangs in installed scripts should be correct to avoid problems when a different
version of Python is set as main active version of Python. If given package does
not support some versions of Python and build system installs scripts with too
generic shebangs, then python_convert_shebangs() should be called to convert
shebangs. This function requires Python version and files / directories. Directories
can be passed only with -r / --recursive option.

python_mod_optimize() is used to compile and optimize Python modules in
pkg_postinst() phase. python_mod_cleanup() is used to remove compiled and
optimized Python modules in pkg_postrm(). Ebuilds, which use distutils.eclass and
install Python modules into site-packages directories, usually do not need to
directly call python_mod_optimize() or python_mod_cleanup(). Paths of modules
installed into site-packages directories should be relative to site-packages
directories. Other paths should be relative to ${ROOT}. python_mod_cleanup()
removes empty directories after cleaning up .py files.

distutils_src_compile(), distutils_src_test() and
distutils_src_install() internally perform actions appropriate for given
type of package. In ebuilds supporting installation for multiple versions of Python,
they define some functions and pass their names to python_execute_function().

If the ebuild name (in ${PN}) differs from the directory created by the package
in site-packages/, then ebuild should define a variable
PYTHON_MODNAME variable to tell distutils_pkg_postinst()
and distutils_pkg_postrm() paths of Python modules.

Important:
For setuptools-0.6a9 and newer you no longer have to remove _require options
other than tests_require because starting with this version
--single-version-externally-managed is made automatic when --root is used which
solves the problem. The new distutils_src_unpack function handles
use_setuptools() problems. The methods explained in this section - i.e. removing
_requires and use_setuptools() with sed - shouldn't be used anymore.

Packages that use setuptools to install use _require options like
tests_require,install_require,setup_requires in setup.py. These are nice to
learn about dependencies but you don't want them in setup.py when you're
installing the package. The following is from the setuptools
homepage section on setup_requires:

A string or list of strings specifying what other distributions need to be
present in order for the setup script to run. setuptools will attempt to obtain
these (even going so far as to download them using EasyInstall) before
processing the rest of the setup script or commands.

—setuptools developer's guide

We have lovely package managers which can download stuff for us and verify their
digests thus we don't want to download any packages using EasyInstall. There are
other options like tests_require, install_requires that behave the same way.

Some packages have a ez_setup.py along with the usual setup.py. This is a python
script to download and install appropriate setuptools. To do this
use_setuptools() is called from ez_setup.py before importing setuptools.

Code Listing 3.1: use_setuptools() from ez_setup.py

defuse_setuptools(
version=DEFAULT_VERSION, download_base=DEFAULT_URL, to_dir=os.curdir,
download_delay=15
):
"""Automatically find/download setuptools and make it available on sys.path
[...]

Just like the _require options, if a setup.py script calls use_setuptools() from
ez_setup.py you should remove it. Below is an example which illustrates how to
do it.

When testing python packages it's important to make sure we're actually testing
the package that is going to be merged not the already installed package. We can
solve the problem by setting the PYTHONPATH environment variable which augments
the default search path for module files. Here are two examples:

repoman may issue a warning saying dev-python/setuptools is a suspicious RDEPEND. Note however that setuptools is quite often a run-time dependency by code that installs commands in /usr/bin, uses pkg_resources to require specific package versions or makes use of entry points in a plugin system.

If you emerge a package that uses setuptools and it installs commands in /usr/bin you can look at those commands and easily determine if setuptools is required at run-time.

Summary:
This guide is supposed to be a help for (new) maintainers of Python packages.
Besides valuable hints and tricks, there are guidelines for version bumps and
drops, stabilization, correct eclass usage, correct dependencies and tests.