This site uses cookies to deliver our services and to show you relevant ads and job listings.
By using our site, you acknowledge that you have read and understand our Cookie Policy, Privacy Policy, and our Terms of Service.
Your use of Stack Overflow’s Products and Services, including the Stack Overflow Network, is subject to these policies and terms.

Join us in building a kind, collaborative learning community via our updated
Code of Conduct.

And to preempt the close votes, I felt this was a more general question than stackoverflow.com/questions/29950300/… , and so I didn't feel comfortable editing that question or posting an overly general answer on that post.
– FlimmJan 10 '17 at 16:33

2 Answers
2

PyPI packages not in the standard library:

virtualenv is a very popular tool that creates isolated Python environments for Python libraries. If you're not familiar with this tool, I highly recommend learning it, as it is a very useful tool, and I'll be making comparisons to it for the rest of this answer.

It works by installing a bunch of files in a directory (eg: env/), and then modifying the PATH environment variable to prefix it with a custom bin directory (eg: env/bin/). An exact copy of the python or python3 binary is placed in this directory, but Python is programmed to look for libraries relative to its path first, in the environment directory. It's not part of Python's standard library, but is officially blessed by the PyPA (Python Packaging Authority). Once activated, you can install packages in the virtual environment using pip.

pyenv is used to isolate Python versions. For example, you may want to test your code against Python 2.6, 2.7, 3.3, 3.4 and 3.5, so you'll need a way to switch between them. Once activated, it prefixes the PATH environment variable with ~/.pyenv/shims, where there are special files matching the Python commands (python, pip). These are not copies of the Python-shipped commands; they are special scripts that decide on the fly which version of Python to run based on the PYENV_VERSION environment variable, or the .python-version file, or the ~/.pyenv/version file. pyenv also makes the process of downloading and installing multiple Python versions easier, using the command pyenv install.

pyenv-virtualenv is a plugin for pyenv by the same author as pyenv, to allow you to use pyenv and virtualenv at the same time conveniently. However, if you're using Python 3.3 or later, pyenv-virtualenv will try to run python -m venv if it is available, instead of virtualenv. You can use virtualenv and pyenv together without pyenv-virtualenv, if you don't want the convenience features.

virtualenvwrapper is a set of extensions to virtualenv (see docs). It gives you commands like mkvirtualenv, lssitepackages, and especially workon for switching between different virtualenv directories. This tool is especially useful if you want multiple virtualenv directories.

pyenv-virtualenvwrapper is a plugin for pyenv by the same author as pyenv, to conveniently integrate virtualenvwrapper into pyenv.

pipenv, by Kenneth Reitz (the author of requests), is the newest project in this list. It aims to combine Pipfile, pip and virtualenv into one command on the command-line. The virtualenv directory typically gets placed in ~/.local/share/virtualenvs/XXX, with XXX being a hash of the path of the project directory. This is different from virtualenv, where the directory is typically in the current working directory.

The Python Packaging Guide recommends pipenv when developing
Python applications (as opposed to libraries). There does not seem to
be any plans to support venv instead of virtualenv (#15).
Confusingly, its command-line option --venv refers to the
virtualenv directory, not venv, and similarly, the environment
variable PIPENV_VENV_IN_PROJECT affects the location of the
virtualenv directory, not venv directory (#1919).

Standard library:

pyvenv is a script shipped with Python 3 but deprecated in Python 3.6 as it had problems (not to mention the confusing name). In Python 3.6+, the exact equivalent is python3 -m venv.

venv is a package shipped with Python 3, which you can run using python3 -m venv (although for some reason some distros separate it out into a separate distro package, such as python3-venv on Ubuntu/Debian). It serves a similar purpose to virtualenv, and works in a very similar way, but it doesn't need to copy Python binaries around (except on Windows). Use this if you don't need to support Python 2. At the time of writing, the Python community seems to be happy with virtualenv and I haven't heard much talk of venv.

Most of these tools complement each other. For instance, pipenv integrates pip, virtualenv and even pyenv if desired. The only tools that are true alternatives to each other here are venv and virtualenv.

Recommendation for beginners:

This is my personal recommendation for beginners: start by learning virtualenv and pip, tools which work with both Python 2 and 3 and in a variety of situations, and pick up the other tools once you start needing them.

I love it when people answer questions I didn't even know I had.
– skrrgwasmeJan 10 '17 at 16:32

22

This is very helpful! So why are there 8 tangled things instead of 1? (“There should be one – and preferably only one – obvious way to do it.” -- The Zen of Python)
– Jerry101Apr 10 '17 at 19:14

21

@Jerry101, the introduction of venv is in part a response to that mess. If you want to help improve the situation, I suggest you use venv and encourage others to do the same.
– Magnus Lind OxlundMay 14 '17 at 18:35

6

"the introduction of venv is in part a response to that mess" How come when there are too many things that do 'something like X', people always think they can improve that mess by making an other thing that does 'something like X'. Its kind of funny actually. We are now 4 years later... so may be pertinent to ask, did venv actually solve that problem?
– KrisMay 26 '17 at 0:24

16

The only two tools on the list that truly cover what is arguably the same territory are virtualenv and venv, so the characterization that we're dealing with a mess caused by several competing tools is not very precise. The list does, however, consist of several virtual environment-related tools, all with similar-sounding names. That can be confusing, especially to users who are just learning about them. Did venv improve the situation? It did offer a more light-weight alternative to other virtual environment tools, benefiting from native modifications and a spot in the standard library. …
– Magnus Lind OxlundMay 29 '17 at 17:48

I would just avoid the use of virtualenv after Python3.3+ and instead use the standard shipped library venv. To create a new virtual environment you would type:

$ python3 -m venv <MYVENV>

virtualenv tries to copy the Python binary into the virtual environment's bin directory. However it does not update library file links embedded into that binary, so if you build Python from source into a non-system directory with relative path names, the Python binary breaks. Since this is how you make a copy distributable Python, it is a big flaw. BTW to inspect embedded library file links on OS X, use otool. For example from within your virtual environment, type:

$ otool -L bin/python
python:
@executable_path/../Python (compatibility version 3.4.0, current version 3.4.0)
/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1238.0.0)

Consequently I would avoid virtualenvwrapper and pipenv. pyvenv is deprecated. pyenv seems to be used often where virtualenv is used but I would stay away from it also since I think venv also does what pyenv is built for.

venv creates virtual environments in the shell that are fresh and sandboxed, with user-installable libraries, and it's multi-python safe. Fresh because virtual environments only start with the standard libraries that ship with python, you have to install any other libraries all over again with pip install while the virtual environment is active. Sandboxed because none of these new library installs are visible outside the virtual environment, so you can delete the whole environment and start again without worrying about impacting your base python install. User-installable libraries because the virtual environment's target folder is created without sudo in some directory you already own, so you won't need sudo permissions to install libraries into it. Finally it is multi-python safe, since when virtual environments activate, the shell only sees the python version (3.4, 3.5 etc.) that was used to build that virtual environment.

pyenv is similar to venv in that it lets you manage multiple python environments. However with pyenv you can't conveniently rollback library installs to some start state and you will likely need admin privileges at some point to update libraries. So I think it is also best to use venv.

In the last couple of years I have found many problems in build systems (emacs packages, python standalone application builders, installers...) that ultimately come down to issues with virtualenv. I think python will be a better platform when we eliminate this additional option and only use venv.

Except that venv appears to be missing fundamental elements (such as an analogous to add2virtualenv), making import management a pain. And, good luck trying to help using it on SO. I tried switching to venv; It has been so painful that I'm switching back.
– SteveJJan 7 at 3:47

2

add2virtualenv tweaks your PYTHONPATH by adding a custom _virtualenv_path_extensions.pth file under site-packages. Alternatively you could update the PYTHONPATH environment variable in the bin/activate file which you call every time you activate the virtual environment. Or you could add symlinks under site-packages to point to the extra directories. Both of these alternatives are more transparent to the traditional command line tools developers widely use to troubleshoot. The use of a custom .pth with an undocumented name, makes it seem more magical IMO.
– Riaz RizviJan 8 at 17:36

Tried adding to PYTHONPATH in bin/activate -- no luck; may try the others - thanks for the suggestion. Trouble is, I've read the docs thoroughly and couldn't find anything on it. I posted on SO, nobody could seem to help. It would be easy for this to turn into a discussion (I think it would be a good one), but for now -- my problem with recommending venv for general use is that for us mere python mortals, it doesn't seem ready for prime-time without a tool equivalent to venvwrapper. I can easily tell someone to type 'add2virtualenv', sending them to site-packages, not so much.
– SteveJJan 8 at 18:04

4

Okay so I've confirmed on stackoverflow.com/questions/48130371/… that a correct update to PYTHONPATH obviates the need for add2virtualenv. Regarding lack of help on SO from your first comment, my only suggestion is upvote answers if they fix your problem, to motivate people to troubleshoot for you when you post? A half an hour of investigation+write-up in exchange for a mouse click? Sounds like a good trade...
– Riaz RizviJan 9 at 17:22

3

No, you are right on -- I try to be good about upvoting. Heck, if you were in my area I would buy you a beer. I'll keep good on my promise and see if the python doc folks will let me add the change to /bin/activate the official docs for clarity. Though I'm not great, I'm not awful at python. If it was hard for me... Anyway, thank you for your time - wish you the best.
– SteveJJan 9 at 18:15