Connect other Accounts

User (root/sudo free) installation of Python modules.

For this tutorial I will assume user installation of new packages by building from downloaded source, ie "python setup.py install." While this tutorial also applies to user installations with the easy_install command line tool, I reserve easy_install related notes to the "Gotcha's" section at the end of this tutorial.

This tutorial also assumes the reader is a user rather than a developer of python packages and need not worry about multiple or virtual python installations in day to day use. Of course updating to a different version [2.4,2.5,2.6,...] of python requires the user to rebuild their user library.

A word on python versions: Common python versions include 2.4 (what is on the system administered Linux computers where I work), 2.5 on Mac OS X 10.5 and 2.6 on Mac OS X 10.6. What is below is based on the official 2.6 docs, and the example uses syntax appropriate for version 2.4. It will work for all three versions. A new feature added in python 2.6 that would provide alternatives to this technique is discussed briefly in the "Gotchas" section.

Gist

A user configuration file, ~/.pydistutils.cfg, will override the internal system path for python package installations, redirecting the built libraries (lib), scripts (bin) and data (share) into user owned and specified directories. You must simply tell the python installer where theses directories are located.

The user file, ~/.pydistutils.cfg, has the following lines, using a pretty obvious syntax:

This example assumes you will simply create an alternate "usr" tree in your home (~) directory, mimicking the UNIX /usr directory. There is nothing special about the names used for these directory paths, however. Generally, these user directories need not exist to specify them in ~/.pydistutils.cfg but to be safe one could simply run:

The location of your user Python library directory (install_lib) must be in your Python path ("import sys; print sys.path") prior to using the new modules OR installing new modules that have dependencies on other user installed modules. This requirement can be accomplished by explicitly adding the library directory to your python path. For the example above you would do this:

in bash:

export PYTHONPATH=~/usr/lib/python2.4/site-packages:${PYTHONPATH}

or csh:

setenv PYTHONPATH ~/usr/lib/python2.4/site-packages:${PYTHONPATH}

where PYTHONPATH is ordered such that earlier entries supersede latter ones and the entire $PYTHONPATH supercedes all system libraries (but see Gotcha's).

More details

To use the scripts one might want to add the install_bin directory to your unix path, $PATH. Else one would need to explicitly call the above example as ~/usr/bin/tinyurl ..., instead.

Gotchas

1. Regarding default user writable installation directories: Mac

In some cases the current system python knows to check for a default user writable directory for modules in user installed packages without resorting to the $PYTHONPATH variable. This is the case for Mac OS X (Leopard), where the directory:

~/Library/Python/2.5/site-packages

is automatically searched for modules to add to the python path (whether you knew that or not might be another point of value for this tutorial!). You would instead substitute that directory,

install_lib = ~/Library/Python/2.5/site-packages

in ~/.pydistutils.cfg. The gotcha for this is that this directory is searched *last* on the system python module path and as such the user may not be able to override system installed packages using this particular default directory. You cannot override this path ordering by adding it to your $PYTHONPATH because it is interpreted as a duplicate path and ignored.

2. easy_install

The easy_install command line tool will respect the installation instructions provided in ~/.pydistutils.cfg, however, there are gotchas related to how easy_install manipulates the module path that should be mentioned.

The most critical issue with easy_install and user installed python packages is the fact that packages can be installed in different ways that cause their modules to have priorities that appear to differ from the formal system path. The way the python module path works, $PYTHONPATH should override all other directories (this is built into the basic architecture). The standard path ordering would be this:

This is not intuitive unless you understand that while easy_install follows the same path ordering process, it builds its packages (called "eggs") and pre-pends them to the other paths. The module path now looks like:

easy_install to $PYTHONPATH

easy_install to System owned site-directories

easy_install to ~/Library/Python/2.X/site-packages [on Macs]

$PYTHONPATH

System owned site-directories

~/Library/Python/2.X/site-packages [on Macs]

And your numpy installations are thus ordered:

numpy 1.4 (w/easy_install)

numpy 1.5 svn (with python setup.py install to $PYTHONPATH location)

numpy 1.3 system

on the python module path. Moreover, if your system installation of numpy was built with easy_install your 1.5svn attempt falls to third place above. As such if your system owned installation used easy_install to install packages then you are basically forced to also use easy_install because the system easy_install'ed packages will supersede even your $PYTHONPATH unless you also install with easy_install.

This is not to discount easy_install. I always use it. If you or your root install have used it, just keep always using it. For example, the way I do this with the numpy 1.5 svn versions is:

Using easy_install, also makes it straightforward for me to switch versions of numpy that I have already installed. For the working example, I have an easy_installed numpy 1.4 and numpy 1.5.0.dev8079. To switch I simply run easy_install and point at the requested version:

Starting from scratch with easy_install

At my institution the system administered Linux workstations use python2.4 and lack easy_install and the ipython shell. In this case I set up my ~/.pydistutils.cfg according to the example at the beginning of the tutorial, created the three install directories, and ran:

which installed easy_install into ~/usr/bin. Then, without problem, I then ran

~/usr/bin/easy_install -Z nose

~/usr/bin/easy_install -Z numpy

python -c "import numpy;numpy.test()"

~/usr/bin/easy_install -Z ipython==0.9.1

~/usr/bin/easy_install -Z PIL

~/usr/bin/easy_install -Z matplotlib

3. Python 2.6+

Python 2.6 adds a default tree for user directories that is searched before system libraries and without specifying it on the user $PYTHONPATH. This predefined tree is rooted at "~/.local" in unix environs, where user installed packages are added in "~/.local/lib/python2.6/site-packages" and executable scripts in ~/.local/bin. You can use this directory without ~/.pydistutils.cfg by compiling source packages with:

python setup.py install --user

Note that an entry in ~/.pydistutils.cfg file, such as

install_lib = ~/usr/lib/python2.6/site-packages

will override the --user option, redirecting modules to install_lib. In this specific case "~/usr/lib/python2.6/site-packages" will not be automatically searched for modules unless it is added to $PYTHONPATH. Further, the search path now looks like:

Tags

To suggest a new feed to our aggregator, fill in the Contact Form on the right !

Contribute to AstroPython

Contributions and feedback from the community are strongly encouraged.If you find a bug or wish to suggest new features for the site , file an issue on Github. If you simply have a question or comment for the AstroPython team please fill in this form and we'll get back to you as soon as possible.
If you have a new resource, question, or any other content that you'd like to post to the web site then likewise fill in the creation form. If you wish to have elevated priveleges , you can use the form to contact us.

Feedback | Contribute | Contact

We would love to hear from you but please login to access the Contact Form !

About Us : Team AstroPython

Tom Aldcroft is a Science
Operations Team member and Flight Director for the Chandra X-ray Observatory and works at the Smithsonian Astrophysical
Observatory. He uses Python for Chandra
spacecraft operations analysis as well as research on several X-ray
survey projects. He is the author of the
asciitable,
cosmocalc, and
deproject
packages. He is also
interested in Python for the web and writes Django and Google App Engine applications
for presenting large multi-wavelength survey datasets. He has been the primary mentor for AstroPython

Thomas Robitaille is a Spitzer
postdoctoral fellow at the Harvard-Smithsonian Center for Astrophysics. He works on
studying Galactic star formation, mostly through mid-infrared to mm
wavelengths. This includes modeling multi-wavelength observations of young
stars and studying the global properties of star formation in the
Milky-Way. In his spare time he is a keen Python programmer and is one of
the co-developers of the APLpy and
ATpy packages. He also developed
the IDLSave package.He has been the secondary mentor for the project.

Aman Jhunjhunwala is an undergraduate Computer Science student with a passion for Big Data,Machine Learning and Web technologies. He has been the primary developer of this site as a part of Google Summer of Code-2015 programme under the AstroPy project.