Python2.5 on SLC4

So, a recurring problem when writing django apps, or using django tools is that then often use features only on python2.4 or python2.4 (properties, with: clauses, ...).

Here's a recipe to get python2.5 onto a SLC4 system without causing too much pain. Again it uses pip and virtualenv to do the heavy lifting.

First, install python2.5 - the ATLAS rpms in AFS seem the easiest place (but it's built by the SLC maintainers, so should be available someone more standard too !)

# rpm -Uvh /afs/cern.ch/atlas/offline/external/GRID/ddm/ext/RPMS.py25.x86_64/python25-2.5.2-2.slc4.x86_64.rpm
# which python2.5
/usr/bin/python2.5
# python2.5
Python 2.5.2 (r252:60911, Apr 28 2008, 11:39:35)
[GCC 3.4.6 20060404 (Red Hat 3.4.6-9)] on linux2
Type "help", "copyright", "credits" or "license" for more information.

So now we've got python2.5, let's get virtualenv and pip running. First, get easy_install:

# wget http://peak.telecommunity.com/dist/ez_setup.py
--11:40:46-- http://peak.telecommunity.com/dist/ez_setup.py
=> `ez_setup.py'
Resolving peak.telecommunity.com... 209.190.5.234
Connecting to peak.telecommunity.com|209.190.5.234|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 9,716 (9.5K) [text/plain]

100%[=========================================================================================================>] 9,716 39.37K/s

11:40:47 (39.34 KB/s) - `ez_setup.py' saved [9716/9716]
# python2.5 ez_setup.py --install-dir /usr/lib64/python2.5/site-packages/
Downloading http://pypi.python.org/packages/2.5/s/setuptools/setuptools-0.6c9-py2.5.egg
Processing setuptools-0.6c9-py2.5.egg
Copying setuptools-0.6c9-py2.5.egg to /usr/lib64/python2.5/site-packages
Adding setuptools 0.6c9 to easy-install.pth file
Installing easy_install script to /usr/lib64/python2.5/site-packages/
Installing easy_install-2.5 script to /usr/lib64/python2.5/site-packages/

Installed /usr/lib64/python2.5/site-packages/setuptools-0.6c9-py2.5.egg
Processing dependencies for setuptools==0.6c9
Finished processing dependencies for setuptools==0.6c9
# python2.5 /usr/lib64/python2.5/site-packages/easy_install --install-dir /usr/lib64/python2.5/site-packages/ --script-dir /usr/bin pip
Searching for pip
Reading http://pypi.python.org/simple/pip/
Reading http://pip.openplans.org
Best match: pip 0.3.1
Downloading http://pypi.python.org/packages/source/p/pip/pip-0.3.1.tar.gz#md5=78102ddbb040a183dd361b5d432cdf88
Processing pip-0.3.1.tar.gz
Running pip-0.3.1/setup.py -q bdist_egg --dist-dir /tmp/easy_install-TwXZ62/pip-0.3.1/egg-dist-tmp-UPf9BM
warning: manifest_maker: MANIFEST.in, line 3: 'prune' expects a single
warning: manifest_maker: MANIFEST.in, line 4: 'prune' expects a single
warning: manifest_maker: MANIFEST.in, line 5: 'prune' expects a single
zip_safe flag not set; analyzing archive contents...
pip: module references __file__
Adding pip 0.3.1 to easy-install.pth file
Installing pip script to /usr/bin

Installed /usr/lib64/python2.5/site-packages/pip-0.3.1-py2.5.egg
Processing dependencies for pip
Finished processing dependencies for pip
# python2.5 /usr/lib64/python2.5/site-packages/easy_install --install-dir /usr/lib64/python2.5/site-packages/ --script-dir /usr/bin virtualenv
Searching for virtualenv
Reading http://pypi.python.org/simple/virtualenv/
Best match: virtualenv 1.3.2
Downloading http://pypi.python.org/packages/2.5/v/virtualenv/virtualenv-1.3.2-py2.5.egg#md5=f2cd2b10b8be8b57e74cb1830fc0b504
Processing virtualenv-1.3.2-py2.5.egg
creating /usr/lib64/python2.5/site-packages/virtualenv-1.3.2-py2.5.egg
Extracting virtualenv-1.3.2-py2.5.egg to /usr/lib64/python2.5/site-packages
Adding virtualenv 1.3.2 to easy-install.pth file
Installing virtualenv script to /usr/bin

Installed /usr/lib64/python2.5/site-packages/virtualenv-1.3.2-py2.5.egg
Processing dependencies for virtualenv
Finished processing dependencies for virtualenv

Note the path hacking we had to do to get things all in the right place - EasyInstall doesn't seem to expect site-packages to be in /usr/lib64 on x86_64. Now you've got this going, you can just go ahead and create virtualenv's. Since the /bin/virtualenv and /bin/pip use python2.5, there's not even any need to supply the '-p' option to virtualenv to specify the python to use - it'll default to 2.5.

# pip -E /tmp/devo install Django
Creating new virtualenv environment in /tmp/devo
New python executable in /tmp/devo/bin/python2.5
Also creating executable in /tmp/devo/bin/python
Installing setuptools...done.....
Downloading/unpacking Django
Downloading Django-1.0.2-final.tar.gz (4.6Mb): 4.6Mb downloaded
Running setup.py egg_info for package Django
Installing collected packages: Django
Running setup.py install for Django
changing mode of build/scripts-2.5/django-admin.py from 644 to 755
changing mode of /tmp/devo/bin/django-admin.py to 755
Successfully installed Django
# source /tmp/devo/bin/activate
(devo)[root@lxbra1905 ~]# python
Python 2.5.2 (r252:60911, Apr 28 2008, 11:39:35)
[GCC 3.4.6 20060404 (Red Hat 3.4.6-9)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
/>>> import django
/>>>

python (and Django) development environment

I like virtualenv and pip. Sadly, they don't work so well with python2.3 - virtualenv will work if you manually copy a version of subprocess.py from a python2.4/2.5 machine into site-packages. pip just won't work (it uses @property)...

Anyway, together they make it very easy to deploy a development environment for Django. Here are some notes (using python 2.4 on SLC5):

Starting - install virtualenv and pip
Use easy_install to install virtualenv and pip (a bit of a bootstrap problem - but this is the last time to use easy_install on your OS python directories !)

$ easy_install virtualenv
...
...
$ easy_install pip
Searching for pip
Reading http://pypi.python.org/simple/pip/
Reading http://pip.openplans.org
Best match: pip 0.3.1
Downloading http://pypi.python.org/packages/source/p/pip/pip-0.3.1.tar.gz#md5=78102ddbb040a183dd361b5d432cdf88
...
...
Adding pip 0.3.1 to easy-install.pth file
Installing pip script to /usr/bin

Installed /usr/lib/python2.4/site-packages/pip-0.3.1-py2.4.egg
Processing dependencies for pip
Finished processing dependencies for pip

First example - just installing django with virtualenv
This uses virtualenv to set up a virtual environment, and standard easy_install to install django.

$virtualenv devo
New python executable in devo/bin/python
Installing setuptools.............done.

$easy_install django
Searching for django
Reading http://pypi.python.org/simple/django/
Reading http://www.djangoproject.com/
Reading http://www.djangoproject.com/download/1.0.1-beta-1/tarball/
Best match: Django 1.0.2-final
Downloading http://media.djangoproject.com/releases/1.0.2/Django-1.0.2-final.tar.gz
Processing Django-1.0.2-final.tar.gz
...
...
...
Installing django-admin.py script to /tmp/jamesc/devo/bin

Installed /tmp/jamesc/devo/lib/python2.4/site-packages/Django-1.0.2_final-py2.4.egg
Processing dependencies for django
Finished processing dependencies for django

$. ./devo/bin/activate

(devo)$python
Python 2.4.3 (#1, Mar 13 2008, 13:35:20)
[GCC 4.1.2 20070626 (Red Hat 4.1.2-14)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
h[1] >>> import django
h[1] >>>

Second Example - Doing it with pip
pip can call out to virtualenv to set up the environment it needs, so what is above becomes even easier -

13:45:03[jamesc]vtb-generi$pip install -E devo django
Creating new virtualenv environment in /tmp/jamesc/devo
New python executable in /tmp/jamesc/devo/bin/python
Installing setuptools...done......
Downloading/unpacking django
Downloading Django-1.0.2-final.tar.gz (4.6Mb): 4.6Mb downloaded
Running setup.py egg_info for package django
Installing collected packages: django
Running setup.py install for django
changing mode of build/scripts-2.4/django-admin.py from 644 to 755
changing mode of /tmp/jamesc/devo/bin/django-admin.py to 755
Successfully installed django

Third Example - requirements files in pip
pip has a nice feature of requirement files - here you can put a bunch of packages you want and they get downloaded - either via PyPi or from subversion repositories - this is useful for django add-ons, since many of them are in google code.

Here are two useful requirements file - firstly a generic development tool one : devo-requires.txt

# Devo tools
ipython
pylint
logilab-common
logilab-astng
# Doc tools
docutils
Jinja
Pygments
sphinx
mock

# django + deps
Django
python-ldap
#MySQL-python
pysqlite

And a second with some more specific django tools django-requires.txt

django-pagination
django-reversion
django-gatekeeper
django-batchadmin

# No egg available
-e svn+http://django-command-extensions.googlecode.com/svn/trunk#egg=django-extensions
-e svn+http://django-notification.googlecode.com/svn/trunk#egg=django-notification
# Should use latest SVN version , not package
-e svn+http://django-sphinx.googlecode.com/svn/trunk/#egg=djangosphinx
-e svn+http://django-tagging.googlecode.com/svn/trunk#egg=django-tagging

To install :

$ pip install -E devo -r devo-requires.txt
$ pip install -E devo -r django-requires.txt

You'll need to make sure you have the relevant -devel rpms installed e.g. openldap-devel, sqlite-devel, mysql-devel.

SL5 to SLC5

Vnode only supports SL5 right now - so here's how to turn it into a SLC5 node (for x86_64)

> cd /etc/yum.repos.d/
> rm -f sl-*.repo sl.repo
> rpm -r epel-release
> rpm -Uvh http://linuxsoft.cern.ch/cern/slc5X/i386/yum/updates//yum-conf-5X-6.slc5.cern.noarch.rpm
> yum update

Now for CERN stuff

> yum install cern-wrappers lcm ntp krb5-workstation lpadmincern useraddcern
> lcm --co --all
> chkconfig afs on
> shutdown -r now

Ok, so you should have a slightly more useful box now :)