Thursday, January 21, 2010

Interactive sandboxes: using IPython with virtualenv

A very helpful blog post on IPython and virtualenv by Pedro Algarvio inspired this one. The advice found there takes you 90% to where you want. I'll recap on that 90% but explain and give the extra 10%. I am indebted to Pedro for laying down all the hard work.

First of all, if you are unfamiliar with Ian Bicking's virtualenv package, you should know two things about it:

virtualenv allows you to develop in sane, aseptic, "sandbox" development environments, switch between them seamlessly, and maintain harmonious order in your Python universe.

Ordinarily, IPython, commonly installed system-wide by your preferred package management system, remains oblivious of an activated virtualenv environment, and will just mill about importing packages and modules from the system, rather than the sandbox. This gives two obvious solutions: either 1) configure the system installation of IPython to work with virtualenv, or 2) install IPython in each virtualenv environment. Doug Hellman wrote a nice tutorial on doing the latter approach; here, we'll focus on the former, which I prefer, since it means having to only install IPython once.

IPython (being a Python program) can read and execute Python scripts during launch; we'll use this mechanism to modify IPython's launch to hook into the virtualenv environment we're currently in. First, we'll tell IPython that we want to execute some code in a at startup. If we go to the $HOME/.ipython/ directory, we'll find a file called ipy_user_conf.py. Open the file in your editor of choice, locate the function main(), and at the within that function (I suggest at the end), insert the following line:

execf('~/.ipython/virtualenv.py')

Next, we need to create this file. Still in the $HOME/.ipython/ directory, create a new file called virtualenv.py and open it with your editor. Next, add these contents to this file:

If you took a look at Pedro's version of virtualenv.py, you'll recognize most of his code here. The important difference lies in the trickery we play with sys.path in lines 12 through 22. These lines were inspired by a solution to a problem presented by using site.addsitedir(), which adds new paths only to the end of sys.path.

Adding paths to the end of sys.path has, for our purposes, the undesirable side-effect of allowing system-wide packages and modules to preempt locally installed ones, since Python searches through sys.path for modules and packages in first-to-last order. I have filed a feature request for site.addsitedir() to allow inserting new paths at the beginning of sys.path; in the meantime, we'll use this hack inspired by the modwsgi programmers, which keeps track of the paths before and after the call to site.addsitedir(), then swaps the position of the new paths from the end, to just after the first element, '', which represents the current working directory (which should preempt every other path).

IPython will have access to the contents of the virtualenv sandbox in which you're currently working. For example, if I activate my networkx virtual environment, which has the latest development version of the NetworkX graph library, then fire up IPython, I get the following result (note the line that begins with VIRTUALENV indicating I'm accessing the virtualenv sandbox):

Just that in IPython 0.11 you have to execute the script slightly different: 1. generate a profile by ipython profile create 2. and then add the following to the created ipython_config.py: c.InteractiveShellApp.exec_files = [ "/path/to/the/virtualenv.py" ]

~/.config/ipython/profile_default with IPython 0.13.2/Ubuntu 13.04. Seems to work fine subject to "WARNING: Attempting to work in a virtualenv. If you encounter problems, please install IPython inside the virtualenv."