by Brandon Rhodes • Home

Installing Python packages for Emacs with virtualenv

Date:	25 February 2009
Tags:	computing, emacs, python

Update — my entire Emacs configuration is now public, not just the little snippet that is shown in this old blog post. Simply visit my dot Emacs repository on GitHub at the following URL:
https://github.com/brandon-rhodes/dot-emacs

The only rough edge I have found amidst the otherwise exceptional advice on Ryan McGuire's Enigma Curry blog is that Ryan recommends installing Python packages with:

$ sudo easy_install package_url

This means that his Emacs configuration — which, very generously, he has started maintaining as a project on github so that other people can use it themselves, or branch their own versions — requires root access merely to install. Like Ryan, I also keep my Emacs configuration under version control, so that improvements I check in from one account are easy to check out into all of my other accounts. Although my setup is probably too simple to be interesting as a public project, there is one aspect of it that I should share: unlike Ryan, I use the advanced technology of a virtualenv to hold the Python packages that Emacs needs. The virtual environment lives under my own account, and is easy to create, access, and rebuild, even in the absence of root privileges on a particular machine. Even better, packages that I install or upgrade inside of the virtual environment cannot interfere with Python programs running elsewhere on the system. A “virtual environment” is a little self-contained Python install, whose essential files are copied from your main system Python; no additional compilation is necessary! A virtual environment can be created under any directory on your system. For example, on a Unix system, the default Python probably lives under the /usr directory, and looks like this:

/usr/bin/easy_install             # to add new packages
/usr/bin/python                   # the main executable
/usr/lib/python2.5/               # normal packages
/usr/lib/python2.5/site-packages/ # 3rd party packages

To create a virtual environment where my Emacs-related packages can live in peace by themselves, I simply instruct virtualenv to reproduce a directory structure like the one outlined above, but in a more convenient location — in this case, hidden beneath my ~/.emacs directory:

$ virtualenv ~/.emacs/usr

This means that I now have a little Python directory hierarchy of my own, but writable with my normal user file permissions, and easy to remove and re-create:

/home/bcr/.emacs/usr/bin/easy_install
/home/bcr/.emacs/usr/bin/python
/home/bcr/.emacs/usr/lib/python2.5/
/home/bcr/.emacs/usr/lib/python2.5/site-packages/

Whenever you run the easy_install and python commands that virtualenv has created under this new directory, they will install packages to, and load packages from, the site-packages directory sitting right there in the virtual environment with them. But how, you ask, does the virtualenv command itself get installed? Doesn't that, at least, require root access to easy_install? Not at all! The command is entirely self-contained; just grab the virtualenv.py file from inside of the virutalenv tar archive on its page at the Python Package Index. I always simply include a copy of virtualenv in the files that I push to a new home directory whenever I get a new account somewhere. Here is the real-life build.sh shell script (yes, I am old-fashioned) that I keep in version control along with the .tar.gz archives of the Python packages that I want Emacs to use:

# Brandon's ~/.emacs.d/src/build.sh
# Change to the directory containing this script.

cd $(dirname "$0")

# Clean up directories left over from previous run.

DIRS=(Pymacs-0.23 rope-0.9.2 ropemacs-0.6 pyflakes-0.3.0)
rm -rf ../usr ${DIRS[@]}

# Create the virtualenv.

python2.5 virtualenv.py ../usr
source ../usr/bin/activate

# Unpack the four Python packages that Emacs needs.

tar xvfz Pymacs.tar.gz
tar xvfz rope-0.9.2.tar.gz
tar xvfz ropemacs-0.6.tar.gz
tar xvfz pyflakes-0.3.0.tar.gz

# Install them in "~/.emacs/usr" (the "python2.5" that
# gets called here is the one in the virtualenv, thanks
# to my having sourced its "activate" script up above).

for D in ${DIRS[@]}
do cd $D; python2.5 setup.py install; cd ..; done

# Make fresh pymacs.el symlink from ~/.emacs

rm -f ../pymacs.el
ln -s src/Pymacs-0.23/pymacs.el ../pymacs.el

Without having to assume an elevated privilege even once, this script removes any old virtualenv sitting in the way, builds a new one, and installs the four Python packages into it. As a final step, it creates an easy-to-find symlink to the pymacs.el so that my main Emacs file does not have to remember the name of its source directory, which changes with each new version. My ~/.emacs.d/init.el file simply has to point Emacs at the Python executable inside the virtual environment, and everything works:

;; From Brandon's ~/.emacs.d/init.el
(add-to-list 'load-path "~/.emacs.d")
(setenv "PYMACS_PYTHON" "~/.emacs.d/usr/bin/python2.5")

Actually, the real file is a bit more complicated because several LISP commands are required to set up each feature, but you get the idea: by always running commands from the bin directory of the virtual environment, Emacs can have access to any number of Python packages without my having to touch the system Python install at all! With this one improvement, I have a portable Emacs configuration that can quickly be installed on new machines — which makes it easy whenever Ryan discovers new improvements for my favorite editor! While his original “Emacs as a powerful Python IDE” post was what got me started turning Emacs into a modern IDE, Ryan continues to find new gems from among the many Emacs packages and features floating around on the Internet. I'll be watching!