How to Share Your Python Packages

Overview

Python packages are the building blocks of Python applications. They encapsulate some coherent functionality that can be imported and used by many applications and systems. But first, developers need to find your package and be able to install it. Python provides a free public repository for packages, which is the de facto standard for sharing Python packages. You can also use private package repositories for proprietary packages.

In this tutorial you'll learn how to share your own packages with the community. If you have proprietary packages you need to share just within your company, you will learn how to do that too.

What Is PyPI?

PyPI stands for the Python Package Index. It is a public repository for uploading your packages. Pip is aware of PyPI and can install and/or upgrade packages from PyPI. PyPI used to be called the "Cheese Shop" after Monty Python's famous sketch. If you hear people refer to the "Cheese Shop" in a Python packaging context, don't be alarmed. It's just PyPI.

Prepare a Package for Upload

Before uploading a package, you need to have a package. I'll use the conman package I introduced in the article How to Write Your Own Python Packages. Since PyPI contains thousands of packages, it is very important to be able to describe your package properly if you want people to find it. PyPI supports an impressive set of metadata tags to let people find the right package for the job.

The setup.py file contains a lot of important information used to install your package. But it can also include the metadata used to classify your package on PyPI. Packages are classified using multiple metadata tags. Some of them are textual and some of them have a list of possible values. The full list is available on PyPI's List Classifiers page.

Let's add a few classifiers to setup.py. There is no need to increment the version number as it is only metadata and the code remains the same:

Create an Account

You need to create an account on PyPI to be able to upload packages. Fill in this form and verify your identity by clicking on the URL in the verification email. Now, you need to create a .pypyrc file in your home directory that will contain the information needed to upload packages.

You can add your password too, but it's safer if you don't in case some bad element gets hold of your laptop. This is especially important if you upload popular packages because if someone can upload or upgrade your packages, all the people that use these packages will be vulnerable.

Testing

If you want to test the package registration and upload process and not worry about publishing something incomplete, you can work with the alternative PyPI testing site. Extend your ~/.pypirc file to include a 'pypitest' section.

Remember that the test site is cleaned up regularly, so don't rely on it. It is intended for testing purposes only.

Register Your Package

If this is the first release of your package, you need to register it with PyPI. Twine has a register command, but I can't figure out how to use it. Following the documentation produces an error, and checking the unit tests for twine there is no test for the register command. Oh, well. You can do it manually too using this form to upload the PKG-INFO file. If you use Python 2.7.9+ or Python 3.2+, you can also safely register using python setup.py register.

Let's register conman on the PyPI test site. Note the -r pypitest, which based on the section in ~/.pypirc will register with the test site.

Twine

You can upload a package using python setup.py upload, but it is not secure as it used to send your username and password over HTTP until Python 2.7.9 and Python 3.2. Twine always uses HTTPS and has additional benefits like uploading pre-created distributions, and it supports any packaging format, including wheels. I will use twine for the actual upload.

Twine is not part of the standard library so you need to install it: pip install twine.

Twine uploaded all the distribution formats, both the source and the wheels.

Test Your Package

Once your package is on PyPI, you should make sure you can install it and everything works. Here I create a one-time virtual environment, pip install conman from the PyPI testing site, and then import it. You may want to run more thorough tests for your package.

This specification allows multiple schemes to choose from. I recommend using the popular Semantic Versioning scheme. It is pretty much "<major>.<minor>.<patch>", which corresponds to PEP-440's "<major>.<minor>.<micro>". Just beware of versions continuing the hyphen or plus signs, which are not compatible with PEP-440.

Private Package Repositories

PyPI is great, but sometimes you don't want to share your packages. Many companies and organizations have engineering teams that use Python and need to share packages between them, but are not allowed to share them publicly on PyPI. This is not a problem. You can share packages on private package repositories under your control.

Note that sometimes you may want to have a private package repository under your control just to manage your third-party dependencies. For example, a package author can decide to delete a package from PyPI. If your system relies on being able to install this package from PyPI, you're in trouble.

Devpi

Devpi (which stands for Development Package Index) is a drop-in replacement for the public PyPI server. It is open source and MIT licensed, so you can run it inside your firewall. Devpi is very powerful and has many features that allow it to function as your ultimate packaging server:

Fast PyPI mirror

Uploading, testing and staging with private indexes

Index inheritance

Web interface and search

Replication

Importing/Exporting

Jenkins integration

Devpi has excellent documentation, a plugin system and is in active development with a vibrant community.

Conclusion

Python provides a complete solution for hosting your packages and making them available to your fellow Pythonistas. There is a streamlined process assisted by tools to package and upload packages and make them easy to find and install.

If you need to keep things private, Devpi is here for you as a mature and robust private package repository.