Package management in Python 2 or 3 (Linux and Mac) with virtualenv or venv

Apr 27, 2016

Using this tutorial

This is the basic guide I wish I had starting out with Python package management.

It’s meant to be read beginning to end, but feel free to skip non-applicable sub-bullets and the optional section.

Starts basic and gets you to a spot where you should be comfortable with the most common features.

This tutorial covers Python 2 and 3, Virtualenv and venv, on Linux or Mac.

Written May 1st, 2016 on an HP Stream 11 running Lubuntu 14.04.4 LTS and a MacBook Air 11 mid 2011. Software is always changing, this guide will likely become out-of-date at some point. The software referenced in this tutorial is: pip 8.1.1, virtualenv 15.0.0, Python 2.7.6 and 3.4.3. My terminals run Terminator 0.98 (Lubuntu) and iTerm2 (Mac) both running the Zsh shell with Oh-My-Zsh, but you should be able to run whatever terminal you prefer.

When you see [sudo] in brackets like that, it means that it’s optional depending on your computer’s permissions. If you run a command without sudo but the command requires sudo, you’ll get a permission error. If you need to redo a command with sudo, don’t retype the whole thing after sudo, just type sudo !!. The double exclamation points represent the last terminal command.

Leave out the $ at the beginning of the terminal command examples when you type or copy them into your own terminal. It’s just there to let you know the command is for the terminal.

Periods end most sentences in this guide even if the last part of a sentence is code. Be sure not to copy the period, it is outside the code sample: like this.

Tweet me @snitzr or email me (first name last name gmail) if you have a question or comment. Don’t get stuck.

Basic Q & A

What is a Python package? Python files that you install to give greater functionality to your Python projects.

What is Virtualenv? Virtualenv is a package manager for the Python programming language. It is not a virtual machine running inside your computer (like VirtualBox or Vagrant) and it’s not Docker.

What is a package manager? A package manager lets you keep separate installed Python libraries for each Python program you’re working on.

Why do I need to keep packages for different projects separate? First, you may not need the exact same packages or package versions for each project. Second, if you collaborate, a package manager helps all team members use the exact same packages and versions.

How do I install Python packages? There is a program called pip, which installs Python packages with: $ [sudo] pip install [package-name]. I’ll talk more about installing packages soon.

Is Virtualenv required for development? No, but it should help development go better.

What package manager should I use? Doesn’t Python 3 come with its own package manager? Python 3 has venv. Virtualenv and venv are very similar. I think if you learn one, you will know the other. You can substitute python3 -m venv where virtualenv is mentioned in the tutorial and it should be the same. python3 -m venv can be activated with just, pyvenv, but pyvenv will likely be deprecated in Python 3.6. In the Python 3 install on my Linux machine, the pyvenv script wasn’t even there. Note: I’m talking about pyvenv and not pyenv. Those are two different things with confusingly similar names. Note the “v” in the middle of pyvenv.

Does a virtual environment also hold Python or just Python packages? Just Python packages.

In the same directory as your project, type $ virtualenv ENV. This creates the ENV folder in that directory.

Running $ virtualenv ENV overwrites any existing directory named ENV in the directory you are in.

If you have Python 2 and Python 3 installed, virtualenv defaults to Python 2. If you have Python 3 installed and want to use it in the virtual environment, do the following:

$ which python3, add this output to the next command.

$ virtualenv -p [output from which python3] ENV. The -p flag means “select the following Python version to use within the virtual environment.”

If the above command doesn’t work, substitute python3 -m venv where you see virtualenv in this tutorial.

There is no requirement that dictates the name or capitalization for the virtualenv directory, other than individual style. You can call the directory almost any name you like. ENV and venv seem the most common.

If you use version control software (like Git), you might like to add “ENV/” to your .gitignore or other similar ignore file.

The virtual environment is created but not yet active. Wait to install Python packages specific to your project until you start the virtual environment.

Start the virtual environment

While in the directory with ENV (not inENV, but the directory withENV), run $ source ENV/bin/activate. This activates the virtual environment.

You’ll probably see the name of the virtual environment in parentheses in your terminal. For example: (ENV) foo:~/$, or something like that, it varies with the terminal you use. This helps you know the environment is active. The name in parenthesis is the name of the folder you created using $ virtualenv ENV.

It would be more intuitive if activating the environment was something like, $ virtualenv activate, but that’s not the case (even though deactivating a virtual environment is just the word deactivate, as you’ll see in a moment). If you like, you could add an alias in your terminal setup dotfile like: alias activate="source ENV/bin/activate".

source is sometimes written as just . in other tutorials. They do the same thing.

Install packages

While the virtualenv is running (because you ran $ source ENV/bin/activate), use pip to install packages.

Example: (ENV) $ pip install ipython.

Adding sudo is not needed or recommended for pip inside an active virtual environment.

While virtualenv is running, you will only have access to the packages you installed while running the virtual environment (and vice versa).

Now you can work on your project in this environment, you’re all set.

Most packages you install with pip while running the virtual machine are available for import in your project. For example: if you run pip install fibo, for Fibonacci math functionality, you add the line import fibo into your Python script and now you have access to that module’s functionality and namespace in your program. Here’s Python’s more detailed explanation of importing packages and modules.

Stop the virtual environment

Deactivate the virtualenv with (ENV) $ deactivate. Virtualenv also stops if you close the terminal window or session. Remember to reactivate it with $ source ENV/bin/activate when you want to use it again.

Sharing environments with a friend

$ source ENV/bin/activate in your project directory.

(ENV) $ pip freeze, this is a list of what you’ve installed with pip.

If you installed ipython from the example in this tutorial, you will notice more packages listed than just ipython, this is OK and covered in the next section.

(ENV) $ pip freeze > requirements.txt, create a file named requirements.txt with a list of what you’ve installed with pip.

The name requirements.txt is just the commonly used name, it’s not a rule to name it that.

This is a manual process. Packages installed after this will not appear automatically.

Send a copy of your requirements.txt file to a friend.

Have your friend install pip and virtualenv, if not already installed and up-to-date.

Now you and your friend should have the same Python packages installed for the project you’re working on together.

How did this get in my requirements.txt?

Some packages need to install other packages called “dependencies.” Dependencies are Python packages and show up in the requirements.txt file. But the dependencies might have a name nothing to do with the package they came from. There is no indication in requirements.txt to show what package installed the dependency.

Example: the ipython package installs nine dependencies with names like: backports.shutil-get-terminal-size, pexpect, and traitlets. Suddenly your requirements.txt file is cluttered. If you need to know what package installed the dependencies, the only solution I know is to use (ENV) $ pip show ipython to get a list of what ipython requires. Or, if you’re OK not installing dependencies, install packages with pip’s --no-deps flag: (ENV) $ pip install --no-deps ipython. But --no-deps won’t install a usable ipython without dependencies. I’m only using ipython as an example because it’s a package that installs dependencies, you don’t need to install ipython unless you need it.

Optional: if you need to move the ENV location

This breaks links to ENV and prevents the virtual environment from starting up. You have three options: