Changing Python search path - for good. Or: the magic of the 'site' package.

After using Python for about two years now and being a somewhat active developer, I still frequently run into problems with my Python search path.
Luckily, I usually have root on the boxes I work on, so I could do some hacks.

At the moment, I'm at the IST Austria, where I can use the cluster, but I don't have root. So this time I needed a real solution.

Here it goes:
The problem is the following: I have some locally installed packages, like scikit-learn and joblib, that are not globally installed. That would be easy enough to solve by setting the PYTHONPATH environment variable in my .profile, pointing to the install location.

But I also have newer versions of already installed packages, like IPython. Here, modifying the Python path environment variable doesn't help, as this is appended to the search path. A somewhat hacky solution is to insert your package dir into the search path at the beginning of each script, like so:
import sys
sys.path.insert(0, "MYPATH")
That is somewhat nasty, as you have to insert it into every file. Also, I found it gives you trouble when trying to use parallel computing.

So here is the REAL solution:
*drumroll*
Check out the site package. It tells you how to configure "sites", which are places where Python looks for packages.
Additionally, "site" directories can hold ".pth" files. Those are files that tell Python where to look for additional packages.
You can add sites by using site.addsidedir, but each user also has a standard site, on Linux it is 
~/.local/lib/pythonX.Y/site-packages
You can also get this dir by looking at site.USER_SITE What I did was create a link at 
~/.local/lib/pythonX.Y/site-packages
to point at my local install location.

As not all my packages are installed there (the things that are git checkouts and build inplace), I also added a .pth file there, that points to my other directories.

To make this a bit more concrete:

My locally installed packages are in ~/python_packages/lib/python2.6/site-packages/. So I 
$ ln -s ~/.local/lib/python2.6/site-packages/ ~/python_packages/lib/python2.6/site-packages/
And created a new .pthfile 
~/python_packages/lib/python2.6/site-packages/local.pth
containing: 
import sys; sys.__plen = len(sys.path)
/clusterhome/amueller/checkout/joblib
/clusterhome/amueller/checkout/scikit-learn
./IPython
import sys; new=sys.path[sys.__plen:]; del sys.path[sys.__plen:]; \
p=getattr(sys,'__egginsert',0); sys.path[p:p]=new; sys.__egginsert = p+len(new)
The first and last line are taken from the easy_install.pth and they basically insert the path to the front. (Only lines starting with import are executed in .pth files The other lines add directories to the path. You can check whether you where successful in IPython: 
In [1]: import sys

In [2]: sys.path
Out[2]: 
['',
 '/clusterhome/amueller/python_packages/bin',
 '/clusterhome/amueller/.local/lib/python2.6/site-packages/pyflakes-0.5.0-py2.6.egg',
 '/clusterhome/amueller/checkout/joblib',
 '/clusterhome/amueller/checkout/scikit-learn',
 '/clusterhome/amueller/.local/lib/python2.6/site-packages/IPython',
 '/usr/local/lib/python2.6/dist-packages/scikit_learn-0.9-py2.6-linux-x86_64.egg',
 '/usr/lib/python2.6',
 '/usr/lib/python2.6/plat-linux2',
 '/usr/lib/python2.6/lib-tk',
 '/usr/lib/python2.6/lib-old',
 '/usr/lib/python2.6/lib-dynload',
 '/clusterhome/amueller/.local/lib/python2.6/site-packages',
 '/usr/local/lib/python2.6/dist-packages',
 '/usr/lib/python2.6/dist-packages',
 '/usr/lib/python2.6/dist-packages/PIL',
 '/usr/lib/pymodules/python2.6',
 '/usr/lib/pymodules/python2.6/gtk-2.0',
 '/usr/lib/python2.6/dist-packages/wx-2.8-gtk2-unicode',
 '/clusterhome/amueller/.local/lib/python2.6/site-packages/IPython/extensions']
Success :) Hope that helped any one!

Comments

  1. Leonardo UiedaMay 6, 2012 at 5:37 PM

    Why not use virtualenv instead?

    ReplyDelete
  2. Andreas MuellerMay 6, 2012 at 5:42 PM

    I feel virtualenv is somewhat overkill for this simple problem. I just want one environment, which is mine.

    ReplyDelete
    Replies
    1. UnknownMay 6, 2012 at 10:23 PM

      That is true. But wouldn't a virtualenv be easier to backup and transfer to other machines?

      Delete
    2. Andreas MuellerMay 7, 2012 at 11:25 PM

      Yeah, probably.

      Delete
  3. AnonymousMay 7, 2012 at 11:17 PM

    I think using virtualenv has a lot of advantages.

    The first on top of my head would be the ability to replicate your environment very quickly, using a pip freeze to get all the packages, and then upgrade only the one you need to try, without loosing your current working environment.

    Virtualenwrapper makes also super easy to create a new virtualenv, and switch between them.

    However you knew the package already, so maybe I'm missing something here!

    ReplyDelete
    Replies
    1. Andreas MuellerMay 7, 2012 at 11:22 PM

      Maybe I should use virtualenv. To be honest, I never really used it. I try to keep things simple and just adding some directories to my search path was all I wanted ;)

      Delete

Post a Comment

Popular posts from this blog

Machine Learning Cheat Sheet (for scikit-learn)

Image
As you hopefully have heard, we at scikit-learn are doing a user survey (which is still open by the way). One of the requests there was to provide some sort of flow chart on how to do machine learning. As this is clearly impossible, I went to work straight away. This is the result: [edit2] clarification: With ensemble classifiers and ensemble regressors I mean random forests , extremely randomized trees, gradient boosted trees , and the soon-to-be-come weight boosted trees (adaboost). [/edit2] Needless to say, this sheet is completely authoritative.
Read more

A Wordcloud in Python

Image
Last week I was at Pycon DE , the German Python conference. After hacking on scikit-learn a lot last week, I decided to to something different on my way back, that I had planned for quite a while: doing a wordl -like word cloud . I know, word clouds are a bit out of style but I kind of like them any way. My motivation to think about word clouds was that I thought these could be combined with topic-models to give somewhat more interesting visualizations. So I looked around to find a nice open-source implementation of word-clouds ... only to find none. (This has been a while, maybe it has changed since). While I was bored in the train last week, I came up with this code . A little today-themed taste:
Read more

MNIST for ever....

Image
[update] This post is a bit old, but many people still seem interested. So just a short update: Nowadays I would use Python and scikit-learn to do this. Here is an example of how to do cross-validation for SVMs in scikit-learn. Scikit-learn even downloads MNIST for you. [/update] MNIST is, for better or worse, one of the standard benchmarks for machine learning and is also widely used in then neural networks community as a toy vision problem. Just for the unlikely case that anyone is not familiar with it: It is a dataset of handwritten digits, 0-9, in black on white background. It looks something like this: There are 60000 training and 10000 test images, each 28x28 gray scale. There are roughly the same number of examples of each category in the test and training datasets. I used it in some papers myself even though there are some reasons why it is a little weird. Some not-so-obvious (or maybe they are) facts are: - The images actually contain a 20x20 patch of digi...
Read more

Python things you never need: Empty lambda functions

Ever wanted to create an empty Python function in-line? No? Me neither. But my coworker does... We found the answer: <pre> x = lambda : None </pre> If you ever needed this, please tell me ;) [Edit] Apparently some people do need it :) Btw, most people seem to try Most people try lambda: pass first. But you have to remember, the thing at the right side of the colon is the return value of the function, not the body! So an expression is needed, not a statement. [\Edit]
Read more