Doing the needful...

Giving structure to your monitoring [Part 1: Local and remote monitoring]

2011-01-11T07:55:00.000-08:00

I've spent the last couple of years at CERN working on building monitoring infrastructures for grid systems. Recently a colleague asked about a taxonomy of monitoring and I dredged up the original taxonomy [PDF] we came up with three years ago. It seems to have stood the test of time and I think is still useful as a way to structure your metrics and decide which tool should gather, store and process the metrics. I want to revisit some of the concepts raised there

Local and remote monitoring

The first concept examined in the paper was that of local and remote monitoring systems. It was logical then to split monitoring into local and remote, meaning either on the site/data center where the services resided, or external to it. In 2007 mostly the monitoring services were local, integrated into the fabric, for example Nagios, ganglia, cacti, .... The exception tended to be lightweight 'ping' style services for HTTP. Now with the rise of cloud I see a shift in this - some PaaS monitoring offerings like Circonus are basically by default remote, with the ability to add a local 'enterprise monitoring' component. This makes sense especially for things you want to do analytics or complex event processing on.

A nice split emerges in such a system where the local monitoring component is really responsible for the operation of the underlying infrastructure and the remote monitoring provides rich information on the performance of the application, often in terms of components that the business as well as operations understand. Again in the cloud this can be supplied by the IaaS provider - the acquisition of CloudKick by Rackspace points to the IaaS players trying to build out expertise and services in this area. Amazon is also trying to grow here with their recent offering of free basic CloudWatch monitoring along with alarms on top of the monitoring metrics.

The new players

Since 2007, new open-source monitoring offerings have also come into the market - often as offshoots from internal projects. They include:

Reconnoiter (rebundled as a PaaS offering as Circonus, and also part of the Cloudkick infrastructure). Based on the experience of OmniTI of running super large websites
Graphite. Scalable storage and display of performance metrics
OpenTSDB. Open sourced version of StumbleUpons internal monitoring systems. Built on top of HBase and able to scale to billions of data points.

These are interesting as they focus more on the higher level monitoring functions such as trending and fault detection by complex correlations of metrics as well as moving past RRD for display and storage of performance data. There's a lot of work here still to do for these projects - documentation is sparse and packaging and integration in distributions is weak, but they are looking in interesting directions for solving the storage, visualisation and analysis of metrics.

I'll follow up in separate postings on some of the other areas we looked at - status and performance metrics and categorization of metrics.

Resolutions for 2011

2011-01-10T06:58:00.000-08:00

2011 is set to be a year of big change for me as I leave CERN after 8 years. So I thought it was a good year to write down some resolutions (for the first time ever!). I think it's going to be important for me not to be too mono-focussed on any one area in particular this year, so some resolutions for Body, Speech and Mind:

Body

In general I need to work a bit more on mountain fitness - I ran 1:42 for a half marathon this year, it'd be nice to get the training in to pull that under 1:40

Run 1080 Kilometres (I did ~950 last year)
Enter 7 races (I did 3 last year)
Complete 21 ski tours (~3 last year)
21 other 'mountain days' - trail running, climbing, hiking

Speech

This year I want to write more, both code and text. Especially since I won't have morning coffee colleagues to talk to (other than my cat!), I'm going to try and publish more stuff on this blog and twitter..

49 blog entries
108 tweets
7 new open-source projects on github

Mind

Learn a new language (tibetan, hindi ?) to basic conversational level
Learn a new computer language (Erlang ?)

Let's see in 12 months how I did...

4 years of development activity in under 2 minutes

2011-01-06T04:32:00.001-08:00

We started the WLCG Grid Monitoring Subversion repository in June 2007. For nearly four years it's been the home of the majority of the monitoring tools used to monitor the WLCG infrastructure. Now courtesy of gource here's a 2 minute visualisation of all the checkins over that period.

Stats: 38 commiters, 4898 commits, 450K LoC....

WLCG Monitoring SVN repository visualisation from James Casey on Vimeo.

New Activemq yum repository (and 5.4.2 rpms)

2011-01-06T04:10:00.000-08:00

I got around to setting up a new yum repository to store my activemq rpms.

You can find it at http://packages.platform14.net/repo/activemq/. Currently I've got RHEL/Centos5 packages there. I'll put up a koji instance there soon and add Fedora13/Fedora14 package too. they all build out of my github repository : https://github.com/jamesc/apache-activemq

To enable, here's a .repo file:

[activemq-centos]
name=activemq-centos
failovermethod=priority
baseurl=http://packages.platform14.net/repo/activemq/centos/5/$basearch/
enabled=1
gpgcheck=0

[activemq-source]
name=activemq-source
failovermethod=priority
baseurl=http://packages.platform14.net/repo/activemq/centos/5/SRPMS/
enabled=1
gpgcheck=0

It currently contains the latest 5.4.2 RPMS.

Activemq Rpms

2010-12-19T08:48:00.000-08:00

Due to SSL issues on our koji build server it seems some people can't get at the RPMs.

Anyway, here are direct links for different versions:

for 5.3.2:

http://koji.afroditi.hellasgrid.gr/packages/activemq/5.3.2/3.el5/x86_64/activemq-5.3.2-3.el5.x86_64.rpm

http://koji.afroditi.hellasgrid.gr/packages/activemq/5.3.2/3.el5/x86_64/activemq-client-5.3.2-3.el5.x86_64.rpm

for 5.4.1:

http://koji.afroditi.hellasgrid.gr/packages/activemq/5.4.1/1.el5/x86_64/activemq-5.4.1-1.el5.x86_64.rpm

http://koji.afroditi.hellasgrid.gr/packages/activemq/5.4.1/1.el5/x86_64/activemq-client-5.4.1-1.el5.x86_64.rpm

I'll work on a better solution with a public yum repo for this in the next days.

Fudge in Python

2010-09-14T06:58:00.001-07:00

I though Fudge was an interesting format for messaging when Kirk Wylie first mentioned it last year. Since then C and C++ libraries have appeared alongside the original Java and .Net. Fudge, along with being a strongly-typed, extensible, self-describing format has some nice ideas, such as taxonomies that look like a good fit for some of our messaging use-cases, such as sending GLUE records over messaging. In GLUE, keys are long and a taxomomy would help to reduce the byte count on the wire by a lot. Also the typed nature gives us a big advantage over LDAP, and would save on lots of type conversions.

Another interesting usage (once you have a python implementation) would be automatic translation from Django Models to Fudge messages. This could work very well with django-celery as a custom serializer.

With some time on my hands I decided to take a crack at a Python implementation of the specification. The specification is short, and detailed enough to work from but diving into the reference implementations is highly recommended if you have any doubts on what should be done.

A few days work has lead to a basic implementation up and running with all basic types working (except for date types). There still is a lot missing, including recursive Fudge messages, taxonomies, context objects, streaming interfaces and better accessors for fields in a fudge Message object. A lot of these are the next higher level functionality on top of the basic encoding/decoding functionality.

The code is now up at github, expect changes in the next few days as I flesh out the implementation.

(Some) Java apps run faster on fewer cores...

2010-06-03T11:57:00.001-07:00

From Paul Tyma: Mailinator(tm) Blog: How I sped up my server by a factor of 6

Odd effects of reducing #cores for the JVM to improve message throughput.

ActiveMQ Advisories and STOMP

2010-05-25T06:57:00.003-07:00

Advisory Messages in ActiveMQ are pretty useful, but since I mostly use STOMP clients I never really knew how useful ! I was blocked by this so prompted by a mail from Gary Tully on the activemq-users list and a long weekend coming up I sat down and tried to work out just what was needed. In the end, after a brief diversion into why JmsFrameTranslator isn't actually used in the server, I got a pretty simple implementation up and running. This was aided by XStream already being a dependency and it having pretty good JSON support. Some tests later and I have a first implementation.

The internal openwire DataStructure objects are now marshalled into the STOMP message body as JSON. An example Advosiry sent to /topic/ActiveMQ.Advisory.Consumer.Queue.>:

{"ConsumerInfo": {
  "commandId": 4,
  "responseRequired": true,
  "consumerId": {
    "connectionId": "ID:pb-d-128-141-235-87.cern.ch-52535-1274794978077-4:1",
    "sessionId": 1,
    "value": 1
  },
  "destination": {
    "@class": "ActiveMQQueue",
    "string": "testConsumerAdvisory",
    "null": {}
  },
  "prefetchSize": 1000,
  "maximumPendingMessageLimit": 0,
  "browser": false,
  "dispatchAsync": true,
  "noLocal": false,
  "exclusive": false,
  "retroactive": false,
  "priority": 0,
  "optimizedAcknowledge": false,
  "noRangeAcks": false
}}

The code supports automatically supports all command objects that could be marshalled. One little problem is how the "destination" object is marshalled. Not sure why we get the @class in there.

Producing RPMs for Apache ActiveMQ

2010-05-16T03:32:00.001-07:00

Edit (11-01-21) : See the more recent post on a new YUM repo and ActiveMQ 5.4.2 rpms.

All of the deployment we do is via RPM and a big gap for us has long been the lack of RPMs for FUSE Message Broker/ Apache ActiveMQ.

We've had our own hand-rolled rpms for a while now, and when I saw Dejan working on rpms via the maven plugin for ActiveMQ I offered to submit upstream what we have. It's taken a bit of time to clean it up, but now it's all ready.

Our main deployment platform is Scientific Linux 5, based on RHEL 5. We try where possible to submit things upstream via EPEL, and this means we need to comply with the Fedora packaging guidelines. For Java you must build from source and you can't package dependencies in your rpm, so for a 'real' ActiveMQ rpm I'm going to have to build all the dependencies (and that includes all the camel dependencies too). For such an old OS versions as RHEL5 I think this isn't worth it, but it's on my project list to try and do this for the upcoming RHEL6 release.

Anyway, what does this mean for ActiveMQ rpms now ? Basically they're 'built' from a binary tarball coming from the Apache dist web server. The rpm does some repackaging to make things a bit more linux-like in terms of directory locations and also applies some small patches (AMQ-2726 , AMQ-2727) which I'm waiting to get integrated upstream.

Everything is released for now on github and I hope it gets merged upstream at some point - keep an eye on this bug (AMQ-2105) for progress. If you want to build for yourself, just grab the code and do a make

Built rpms for Centos5/RHEL5/SLC5 are on our build system. Enjoy !

India & Nepal

2010-02-22T02:22:00.001-08:00

I spend 3 weeks this January in Northern India and Nepal, starting in Delhi, going to Bodh Gaya, Varanasi, Kushinagar and Lumbini finishing up in Nepal in Kathmandu. Too few days really, but definitely gave me a taste for the area. Here's some pictures (flickr).

Orderly JSON

2009-12-23T10:27:00.000-08:00

From Simon Willison:

Orderly JSON. Essentially the JSON equivalent of RelaxNG’s compact syntax—a pleasant mini-language for describing JSON objects which compiles to the more verbose JSONSchema format.

Deployment is just a part of dev/ops cooperation, not the whole thing

2009-12-16T06:32:00.000-08:00

Deployment is just a part of dev/ops cooperation, not the whole thing

More details from John Allspaw (flickr ops manager) on how toget dev + ops working together. All nice stuff like better monitoring, allowing developers to easily add monitoring, feature flags and more.

check it out.

New job positions in Europe for Amazon Web Services

2009-12-16T01:30:00.000-08:00

New job positions in Europe for Amazon Web Services
Interesting jobs @AWS including Solutions Architect and Tech Evangelist.

JS bindings for AMQP

2009-12-10T03:42:00.001-08:00

kamaloka-js-0.1.0 AMQP bindings released Makes client side programing easy - seems to also support QMF

Getting started testing in Django

2009-11-29T00:50:00.001-08:00

Lazy Pythonista: Getting Started with Testing in Django : Alex Gaynor provides a good introduction to the simplest (and most
immediately useful) way to test your Django apps - simple view testing.

Information systems from an operational perspective

2009-10-02T04:15:00.000-07:00

Some slides that I presented at EGEE'09 on future ideas for information systems from an operational perspective.

Grid Information systems from an Operations Perspective

View more presentations from jamescasey.

Django slides at GD White Areas

2009-10-02T03:57:00.001-07:00

I gave the following slides today on Django at the White Areas lectures. They give a high level introduction to Django and also show some useful tools for python/Django development.

Introduction to Django

View more presentations from jamescasey.

pip and virtualenv differences (--no-site-packages)

2009-03-18T05:46:00.000-07:00

Just a quick note here found when debugging multiple django installs.

There is a difference between:

> virtualenv foo
> pip install -E foo Django

and

# no virtualenv exists at this point
> pip install -E foo Django

Basically, if pip creates a new virtualenv, it always does it using the '-no-site-packages' option. This isn't so good for things like cx_Oracle, where we already have rpms and it's pretty hard to get it to auto-build on install. Testing this, I ran into yolk - which shows you the list of packages:

# no virtualenv exists at this point
> virtualenv foo
> pip install -E foo yolk
> source foo/bin/activate
(foo)> yolk -l
Python - 2.5.2 - active development (/usr/lib64/python2.5/lib-dynload)
cx_Oracle-py25 - 4.4.1 - active development (/usr/lib64/python2.5/site-packages)
pip - 0.3.1 - active development (/usr/lib64/python2.5/site-packages/pip-0.3.1-py2.5.egg)
setuptools - 0.6c9 - active
setuptools - 0.6c9 - active
setuptools - 0.6c9 - non-active development (/usr/lib64/python2.5/site-packages/setuptools-0.6c9-py2.5.egg)
setuptools - 0.6c9 - non-active development (/usr/lib64/python2.5/site-packages/setuptools-0.6c9-py2.5.egg)
virtualenv - 1.3.3 - active development (/usr/lib64/python2.5/site-packages/virtualenv-1.3.3-py2.5.egg)
wsgiref - 0.1.2 - active development (/usr/lib64/python2.5)
yolk - 0.4.1 - active

Here you can see a cx_Oracle which is inherited correctly from the default environment.

Python2.5 on SLC4

2009-02-18T02:55:00.000-08:00

So, a recurring problem when writing django apps, or using django tools is that then often use features only on python2.4 or python2.4 (properties, with: clauses, ...).

Here's a recipe to get python2.5 onto a SLC4 system without causing too much pain. Again it uses pip and virtualenv to do the heavy lifting.

First, install python2.5 - the ATLAS rpms in AFS seem the easiest place (but it's built by the SLC maintainers, so should be available someone more standard too !)

# rpm -Uvh /afs/cern.ch/atlas/offline/external/GRID/ddm/ext/RPMS.py25.x86_64/python25-2.5.2-2.slc4.x86_64.rpm
# which python2.5
/usr/bin/python2.5
# python2.5
Python 2.5.2 (r252:60911, Apr 28 2008, 11:39:35)
[GCC 3.4.6 20060404 (Red Hat 3.4.6-9)] on linux2
Type "help", "copyright", "credits" or "license" for more information.

So now we've got python2.5, let's get virtualenv and pip running. First, get easy_install:

# wget http://peak.telecommunity.com/dist/ez_setup.py
--11:40:46-- http://peak.telecommunity.com/dist/ez_setup.py
=> `ez_setup.py'
Resolving peak.telecommunity.com... 209.190.5.234
Connecting to peak.telecommunity.com|209.190.5.234|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 9,716 (9.5K) [text/plain]

100%[=========================================================================================================>] 9,716 39.37K/s

11:40:47 (39.34 KB/s) - `ez_setup.py' saved [9716/9716]
# python2.5 ez_setup.py --install-dir /usr/lib64/python2.5/site-packages/
Downloading http://pypi.python.org/packages/2.5/s/setuptools/setuptools-0.6c9-py2.5.egg
Processing setuptools-0.6c9-py2.5.egg
Copying setuptools-0.6c9-py2.5.egg to /usr/lib64/python2.5/site-packages
Adding setuptools 0.6c9 to easy-install.pth file
Installing easy_install script to /usr/lib64/python2.5/site-packages/
Installing easy_install-2.5 script to /usr/lib64/python2.5/site-packages/

Installed /usr/lib64/python2.5/site-packages/setuptools-0.6c9-py2.5.egg
Processing dependencies for setuptools==0.6c9
Finished processing dependencies for setuptools==0.6c9
# python2.5 /usr/lib64/python2.5/site-packages/easy_install --install-dir /usr/lib64/python2.5/site-packages/ --script-dir /usr/bin pip
Searching for pip
Reading http://pypi.python.org/simple/pip/
Reading http://pip.openplans.org
Best match: pip 0.3.1
Downloading http://pypi.python.org/packages/source/p/pip/pip-0.3.1.tar.gz#md5=78102ddbb040a183dd361b5d432cdf88
Processing pip-0.3.1.tar.gz
Running pip-0.3.1/setup.py -q bdist_egg --dist-dir /tmp/easy_install-TwXZ62/pip-0.3.1/egg-dist-tmp-UPf9BM
warning: manifest_maker: MANIFEST.in, line 3: 'prune' expects a single
warning: manifest_maker: MANIFEST.in, line 4: 'prune' expects a single
warning: manifest_maker: MANIFEST.in, line 5: 'prune' expects a single
zip_safe flag not set; analyzing archive contents...
pip: module references __file__
Adding pip 0.3.1 to easy-install.pth file
Installing pip script to /usr/bin

Installed /usr/lib64/python2.5/site-packages/pip-0.3.1-py2.5.egg
Processing dependencies for pip
Finished processing dependencies for pip
# python2.5 /usr/lib64/python2.5/site-packages/easy_install --install-dir /usr/lib64/python2.5/site-packages/ --script-dir /usr/bin virtualenv
Searching for virtualenv
Reading http://pypi.python.org/simple/virtualenv/
Best match: virtualenv 1.3.2
Downloading http://pypi.python.org/packages/2.5/v/virtualenv/virtualenv-1.3.2-py2.5.egg#md5=f2cd2b10b8be8b57e74cb1830fc0b504
Processing virtualenv-1.3.2-py2.5.egg
creating /usr/lib64/python2.5/site-packages/virtualenv-1.3.2-py2.5.egg
Extracting virtualenv-1.3.2-py2.5.egg to /usr/lib64/python2.5/site-packages
Adding virtualenv 1.3.2 to easy-install.pth file
Installing virtualenv script to /usr/bin

Installed /usr/lib64/python2.5/site-packages/virtualenv-1.3.2-py2.5.egg
Processing dependencies for virtualenv
Finished processing dependencies for virtualenv

Note the path hacking we had to do to get things all in the right place - EasyInstall doesn't seem to expect site-packages to be in /usr/lib64 on x86_64. Now you've got this going, you can just go ahead and create virtualenv's. Since the /bin/virtualenv and /bin/pip use python2.5, there's not even any need to supply the '-p' option to virtualenv to specify the python to use - it'll default to 2.5.

# pip -E /tmp/devo install Django
Creating new virtualenv environment in /tmp/devo
New python executable in /tmp/devo/bin/python2.5
Also creating executable in /tmp/devo/bin/python
Installing setuptools...done.....
Downloading/unpacking Django
Downloading Django-1.0.2-final.tar.gz (4.6Mb): 4.6Mb downloaded
Running setup.py egg_info for package Django
Installing collected packages: Django
Running setup.py install for Django
changing mode of build/scripts-2.5/django-admin.py from 644 to 755
changing mode of /tmp/devo/bin/django-admin.py to 755
Successfully installed Django
# source /tmp/devo/bin/activate
(devo)[root@lxbra1905 ~]# python
Python 2.5.2 (r252:60911, Apr 28 2008, 11:39:35)
[GCC 3.4.6 20060404 (Red Hat 3.4.6-9)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
/>>> import django
/>>>

python (and Django) development environment

2009-02-04T04:32:00.000-08:00

I like virtualenv and pip. Sadly, they don't work so well with python2.3 - virtualenv will work if you manually copy a version of subprocess.py from a python2.4/2.5 machine into site-packages. pip just won't work (it uses @property)...

Anyway, together they make it very easy to deploy a development environment for Django. Here are some notes (using python 2.4 on SLC5):

Starting - install virtualenv and pip
Use easy_install to install virtualenv and pip (a bit of a bootstrap problem - but this is the last time to use easy_install on your OS python directories !)

$ easy_install virtualenv
...
...
$ easy_install pip
Searching for pip
Reading http://pypi.python.org/simple/pip/
Reading http://pip.openplans.org
Best match: pip 0.3.1
Downloading http://pypi.python.org/packages/source/p/pip/pip-0.3.1.tar.gz#md5=78102ddbb040a183dd361b5d432cdf88
...
...
Adding pip 0.3.1 to easy-install.pth file
Installing pip script to /usr/bin

Installed /usr/lib/python2.4/site-packages/pip-0.3.1-py2.4.egg
Processing dependencies for pip
Finished processing dependencies for pip

First example - just installing django with virtualenv
This uses virtualenv to set up a virtual environment, and standard easy_install to install django.

$virtualenv devo
New python executable in devo/bin/python
Installing setuptools.............done.

$easy_install django
Searching for django
Reading http://pypi.python.org/simple/django/
Reading http://www.djangoproject.com/
Reading http://www.djangoproject.com/download/1.0.1-beta-1/tarball/
Best match: Django 1.0.2-final
Downloading http://media.djangoproject.com/releases/1.0.2/Django-1.0.2-final.tar.gz
Processing Django-1.0.2-final.tar.gz
...
...
...
Installing django-admin.py script to /tmp/jamesc/devo/bin

Installed /tmp/jamesc/devo/lib/python2.4/site-packages/Django-1.0.2_final-py2.4.egg
Processing dependencies for django
Finished processing dependencies for django

$. ./devo/bin/activate

(devo)$python
Python 2.4.3 (#1, Mar 13 2008, 13:35:20)
[GCC 4.1.2 20070626 (Red Hat 4.1.2-14)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
h[1] >>> import django
h[1] >>>

Second Example - Doing it with pip
pip can call out to virtualenv to set up the environment it needs, so what is above becomes even easier -

13:45:03[jamesc]vtb-generi$pip install -E devo django
Creating new virtualenv environment in /tmp/jamesc/devo
New python executable in /tmp/jamesc/devo/bin/python
Installing setuptools...done......
Downloading/unpacking django
Downloading Django-1.0.2-final.tar.gz (4.6Mb): 4.6Mb downloaded
Running setup.py egg_info for package django
Installing collected packages: django
Running setup.py install for django
changing mode of build/scripts-2.4/django-admin.py from 644 to 755
changing mode of /tmp/jamesc/devo/bin/django-admin.py to 755
Successfully installed django

Third Example - requirements files in pip
pip has a nice feature of requirement files - here you can put a bunch of packages you want and they get downloaded - either via PyPi or from subversion repositories - this is useful for django add-ons, since many of them are in google code.

Here are two useful requirements file - firstly a generic development tool one : devo-requires.txt

# Devo tools
ipython
pylint
logilab-common
logilab-astng
# Doc tools
docutils
Jinja
Pygments
sphinx
mock

# django + deps
Django
python-ldap
#MySQL-python
pysqlite

And a second with some more specific django tools django-requires.txt

django-pagination
django-reversion
django-gatekeeper
django-batchadmin

# No egg available
-e svn+http://django-command-extensions.googlecode.com/svn/trunk#egg=django-extensions
-e svn+http://django-notification.googlecode.com/svn/trunk#egg=django-notification
# Should use latest SVN version , not package
-e svn+http://django-sphinx.googlecode.com/svn/trunk/#egg=djangosphinx
-e svn+http://django-tagging.googlecode.com/svn/trunk#egg=django-tagging

To install :

$ pip install -E devo -r devo-requires.txt
$ pip install -E devo -r django-requires.txt

You'll need to make sure you have the relevant -devel rpms installed e.g. openldap-devel, sqlite-devel, mysql-devel.

SL5 to SLC5

2009-02-03T07:18:00.001-08:00

Vnode only supports SL5 right now - so here's how to turn it into a SLC5 node (for x86_64)

> cd /etc/yum.repos.d/
> rm -f sl-*.repo sl.repo
> rpm -r epel-release
> rpm -Uvh http://linuxsoft.cern.ch/cern/slc5X/i386/yum/updates//yum-conf-5X-6.slc5.cern.noarch.rpm
> yum update

Now for CERN stuff

> yum install cern-wrappers lcm ntp krb5-workstation lpadmincern useraddcern
> lcm --co --all
> chkconfig afs on
> shutdown -r now

Ok, so you should have a slightly more useful box now :)