Orderly JSON

From Simon Willison:

Orderly JSON. Essentially the JSON equivalent of RelaxNG’s compact syntax—a pleasant mini-language for describing JSON objects which compiles to the more verbose JSONSchema format.

Deployment is just a part of dev/ops cooperation, not the whole thing

Deployment is just a part of dev/ops cooperation, not the whole thing

More details from John Allspaw (flickr ops manager) on how toget dev + ops working together. All nice stuff like better monitoring, allowing developers to easily add monitoring, feature flags and more.

check it out.

New job positions in Europe for Amazon Web Services

New job positions in Europe for Amazon Web Services
Interesting jobs @AWS including Solutions Architect and Tech Evangelist.

JS bindings for AMQP

kamaloka-js-0.1.0 AMQP bindings released Makes client side programing easy - seems to also support QMF

Getting started testing in Django

Lazy Pythonista: Getting Started with Testing in Django : Alex Gaynor provides a good introduction to the simplest (and most
immediately useful) way to test your Django apps - simple view testing.

Information systems from an operational perspective

Some slides that I presented at EGEE'09 on future ideas for information systems from an operational perspective.

Grid Information systems from an Operations Perspective

View more presentations from jamescasey.

Django slides at GD White Areas

I gave the following slides today on Django at the White Areas lectures.  They give a high level introduction to Django and also show some useful tools for python/Django development.

Introduction to Django

View more presentations from jamescasey.

pip and virtualenv differences (--no-site-packages)

Just a quick note here found when debugging multiple django installs.

There is a difference between:

> virtualenv foo
> pip install -E foo Django

and

# no virtualenv exists at this point
> pip install -E foo Django

Basically, if pip creates a new virtualenv, it always does it using the '-no-site-packages' option. This isn't so good for things like cx_Oracle, where we already have rpms and it's pretty hard to get it to auto-build on install. Testing this, I ran into yolk - which shows you the list of packages:

# no virtualenv exists at this point
> virtualenv foo
> pip install -E foo yolk
> source foo/bin/activate
(foo)> yolk -l
Python - 2.5.2 - active development (/usr/lib64/python2.5/lib-dynload)
cx_Oracle-py25 - 4.4.1 - active development (/usr/lib64/python2.5/site-packages)
pip - 0.3.1 - active development (/usr/lib64/python2.5/site-packages/pip-0.3.1-py2.5.egg)
setuptools - 0.6c9 - active
setuptools - 0.6c9 - active
setuptools - 0.6c9 - non-active development (/usr/lib64/python2.5/site-packages/setuptools-0.6c9-py2.5.egg)
setuptools - 0.6c9 - non-active development (/usr/lib64/python2.5/site-packages/setuptools-0.6c9-py2.5.egg)
virtualenv - 1.3.3 - active development (/usr/lib64/python2.5/site-packages/virtualenv-1.3.3-py2.5.egg)
wsgiref - 0.1.2 - active development (/usr/lib64/python2.5)
yolk - 0.4.1 - active

Here you can see a cx_Oracle which is inherited correctly from the default environment.

Python2.5 on SLC4

So, a recurring problem when writing django apps, or using django tools is that then often use features only on python2.4 or python2.4 (properties, with: clauses, ...).

Here's a recipe to get python2.5 onto a SLC4 system without causing too much pain. Again it uses pip and virtualenv to do the heavy lifting.

First, install python2.5 - the ATLAS rpms in AFS seem the easiest place (but it's built by the SLC maintainers, so should be available someone more standard too !)

# rpm -Uvh /afs/cern.ch/atlas/offline/external/GRID/ddm/ext/RPMS.py25.x86_64/python25-2.5.2-2.slc4.x86_64.rpm
# which python2.5
/usr/bin/python2.5
# python2.5
Python 2.5.2 (r252:60911, Apr 28 2008, 11:39:35)
[GCC 3.4.6 20060404 (Red Hat 3.4.6-9)] on linux2
Type "help", "copyright", "credits" or "license" for more information.

So now we've got python2.5, let's get virtualenv and pip running. First, get easy_install:

# wget http://peak.telecommunity.com/dist/ez_setup.py
--11:40:46-- http://peak.telecommunity.com/dist/ez_setup.py
=> `ez_setup.py'
Resolving peak.telecommunity.com... 209.190.5.234
Connecting to peak.telecommunity.com|209.190.5.234|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 9,716 (9.5K) [text/plain]

100%[=========================================================================================================>] 9,716 39.37K/s

11:40:47 (39.34 KB/s) - `ez_setup.py' saved [9716/9716]
# python2.5 ez_setup.py --install-dir /usr/lib64/python2.5/site-packages/
Downloading http://pypi.python.org/packages/2.5/s/setuptools/setuptools-0.6c9-py2.5.egg
Processing setuptools-0.6c9-py2.5.egg
Copying setuptools-0.6c9-py2.5.egg to /usr/lib64/python2.5/site-packages
Adding setuptools 0.6c9 to easy-install.pth file
Installing easy_install script to /usr/lib64/python2.5/site-packages/
Installing easy_install-2.5 script to /usr/lib64/python2.5/site-packages/

Installed /usr/lib64/python2.5/site-packages/setuptools-0.6c9-py2.5.egg
Processing dependencies for setuptools==0.6c9
Finished processing dependencies for setuptools==0.6c9
# python2.5 /usr/lib64/python2.5/site-packages/easy_install --install-dir /usr/lib64/python2.5/site-packages/ --script-dir /usr/bin pip
Searching for pip
Reading http://pypi.python.org/simple/pip/
Reading http://pip.openplans.org
Best match: pip 0.3.1
Downloading http://pypi.python.org/packages/source/p/pip/pip-0.3.1.tar.gz#md5=78102ddbb040a183dd361b5d432cdf88
Processing pip-0.3.1.tar.gz
Running pip-0.3.1/setup.py -q bdist_egg --dist-dir /tmp/easy_install-TwXZ62/pip-0.3.1/egg-dist-tmp-UPf9BM
warning: manifest_maker: MANIFEST.in, line 3: 'prune' expects a single
warning: manifest_maker: MANIFEST.in, line 4: 'prune' expects a single
warning: manifest_maker: MANIFEST.in, line 5: 'prune' expects a single
zip_safe flag not set; analyzing archive contents...
pip: module references __file__
Adding pip 0.3.1 to easy-install.pth file
Installing pip script to /usr/bin

Installed /usr/lib64/python2.5/site-packages/pip-0.3.1-py2.5.egg
Processing dependencies for pip
Finished processing dependencies for pip
# python2.5 /usr/lib64/python2.5/site-packages/easy_install --install-dir /usr/lib64/python2.5/site-packages/ --script-dir /usr/bin virtualenv
Searching for virtualenv
Reading http://pypi.python.org/simple/virtualenv/
Best match: virtualenv 1.3.2
Downloading http://pypi.python.org/packages/2.5/v/virtualenv/virtualenv-1.3.2-py2.5.egg#md5=f2cd2b10b8be8b57e74cb1830fc0b504
Processing virtualenv-1.3.2-py2.5.egg
creating /usr/lib64/python2.5/site-packages/virtualenv-1.3.2-py2.5.egg
Extracting virtualenv-1.3.2-py2.5.egg to /usr/lib64/python2.5/site-packages
Adding virtualenv 1.3.2 to easy-install.pth file
Installing virtualenv script to /usr/bin

Installed /usr/lib64/python2.5/site-packages/virtualenv-1.3.2-py2.5.egg
Processing dependencies for virtualenv
Finished processing dependencies for virtualenv

Note the path hacking we had to do to get things all in the right place - EasyInstall doesn't seem to expect site-packages to be in /usr/lib64 on x86_64. Now you've got this going, you can just go ahead and create virtualenv's. Since the /bin/virtualenv and /bin/pip use python2.5, there's not even any need to supply the '-p' option to virtualenv to specify the python to use - it'll default to 2.5.

# pip -E /tmp/devo install Django
Creating new virtualenv environment in /tmp/devo
New python executable in /tmp/devo/bin/python2.5
Also creating executable in /tmp/devo/bin/python
Installing setuptools...done.....
Downloading/unpacking Django
Downloading Django-1.0.2-final.tar.gz (4.6Mb): 4.6Mb downloaded
Running setup.py egg_info for package Django
Installing collected packages: Django
Running setup.py install for Django
changing mode of build/scripts-2.5/django-admin.py from 644 to 755
changing mode of /tmp/devo/bin/django-admin.py to 755
Successfully installed Django
# source /tmp/devo/bin/activate
(devo)[root@lxbra1905 ~]# python
Python 2.5.2 (r252:60911, Apr 28 2008, 11:39:35)
[GCC 3.4.6 20060404 (Red Hat 3.4.6-9)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
/>>> import django
/>>>

python (and Django) development environment

I like virtualenv and pip. Sadly, they don't work so well with python2.3 - virtualenv will work if you manually copy a version of subprocess.py from a python2.4/2.5 machine into site-packages. pip just won't work (it uses @property)...

Anyway, together they make it very easy to deploy a development environment for Django. Here are some notes (using python 2.4 on SLC5):

Starting - install virtualenv and pip
Use easy_install to install virtualenv and pip (a bit of a bootstrap problem - but this is the last time to use easy_install on your OS python directories !)

$ easy_install virtualenv
...
...
$ easy_install pip
Searching for pip
Reading http://pypi.python.org/simple/pip/
Reading http://pip.openplans.org
Best match: pip 0.3.1
Downloading http://pypi.python.org/packages/source/p/pip/pip-0.3.1.tar.gz#md5=78102ddbb040a183dd361b5d432cdf88
...
...
Adding pip 0.3.1 to easy-install.pth file
Installing pip script to /usr/bin

Installed /usr/lib/python2.4/site-packages/pip-0.3.1-py2.4.egg
Processing dependencies for pip
Finished processing dependencies for pip

First example - just installing django with virtualenv
This uses virtualenv to set up a virtual environment, and standard easy_install to install django.

$virtualenv devo
New python executable in devo/bin/python
Installing setuptools.............done.

$easy_install django
Searching for django
Reading http://pypi.python.org/simple/django/
Reading http://www.djangoproject.com/
Reading http://www.djangoproject.com/download/1.0.1-beta-1/tarball/
Best match: Django 1.0.2-final
Downloading http://media.djangoproject.com/releases/1.0.2/Django-1.0.2-final.tar.gz
Processing Django-1.0.2-final.tar.gz
...
...
...
Installing django-admin.py script to /tmp/jamesc/devo/bin

Installed /tmp/jamesc/devo/lib/python2.4/site-packages/Django-1.0.2_final-py2.4.egg
Processing dependencies for django
Finished processing dependencies for django

$. ./devo/bin/activate

(devo)$python
Python 2.4.3 (#1, Mar 13 2008, 13:35:20)
[GCC 4.1.2 20070626 (Red Hat 4.1.2-14)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
h[1] >>> import django
h[1] >>>

Second Example - Doing it with pip
pip can call out to virtualenv to set up the environment it needs, so what is above becomes even easier -

13:45:03[jamesc]vtb-generi$pip install -E devo django
Creating new virtualenv environment in /tmp/jamesc/devo
New python executable in /tmp/jamesc/devo/bin/python
Installing setuptools...done......
Downloading/unpacking django
Downloading Django-1.0.2-final.tar.gz (4.6Mb): 4.6Mb downloaded
Running setup.py egg_info for package django
Installing collected packages: django
Running setup.py install for django
changing mode of build/scripts-2.4/django-admin.py from 644 to 755
changing mode of /tmp/jamesc/devo/bin/django-admin.py to 755
Successfully installed django

Third Example - requirements files in pip
pip has a nice feature of requirement files - here you can put a bunch of packages you want and they get downloaded - either via PyPi or from subversion repositories - this is useful for django add-ons, since many of them are in google code.

Here are two useful requirements file - firstly a generic development tool one : devo-requires.txt

# Devo tools
ipython
pylint
logilab-common
logilab-astng
# Doc tools
docutils
Jinja
Pygments
sphinx
mock

# django + deps
Django
python-ldap
#MySQL-python
pysqlite

And a second with some more specific django tools django-requires.txt

django-pagination
django-reversion
django-gatekeeper
django-batchadmin

# No egg available
-e svn+http://django-command-extensions.googlecode.com/svn/trunk#egg=django-extensions
-e svn+http://django-notification.googlecode.com/svn/trunk#egg=django-notification
# Should use latest SVN version , not package
-e svn+http://django-sphinx.googlecode.com/svn/trunk/#egg=djangosphinx
-e svn+http://django-tagging.googlecode.com/svn/trunk#egg=django-tagging

To install :

$ pip install -E devo -r devo-requires.txt
$ pip install -E devo -r django-requires.txt

You'll need to make sure you have the relevant -devel rpms installed e.g. openldap-devel, sqlite-devel, mysql-devel.

SL5 to SLC5

Vnode only supports SL5 right now - so here's how to turn it into a SLC5 node (for x86_64)

> cd /etc/yum.repos.d/
> rm -f sl-*.repo sl.repo
> rpm -r epel-release
> rpm -Uvh http://linuxsoft.cern.ch/cern/slc5X/i386/yum/updates//yum-conf-5X-6.slc5.cern.noarch.rpm
> yum update

Now for CERN stuff

> yum install cern-wrappers lcm ntp krb5-workstation lpadmincern useraddcern
> lcm --co --all
> chkconfig afs on
> shutdown -r now

Ok, so you should have a slightly more useful box now :)