Ansible Variables all of a Sudden Go Missing?

I’ve written a playbook which deploys a working development environment for some of our internal systems. I’ve tested it with various versions of RHEL. Yet when I ran it against a fresh install of Fedora it failed:

fatal: [192.168.1.233] => {'msg': "One or more undefined variables: 'ansible_lsb' is undefined", 'failed': True}

It turned out, that ansible gets it’s facts through different programs on the remote machine. If some of these programs are not available (in this instance it was lsb_release) the variables are not populated resulting in this error.

So check if all variables you access are indeed available with:

$ ansible -m setup <yourhost>

No such file or directory FakePlugin.egg-info

In case you run into a similar issue which ends up in a traceback like this:

ZopeXMLConfigurationError: File "/opt/works/projects/plone4/nuw.types/nuw/types/configure.zcml", line 12.2-12.37
ZopeXMLConfigurationError: File "/home/roman/.buildout/eggs/mooball.plone.activecampaign-0.2-py2.7.egg/mooball/plone/activecampaign/configure.zcml", line 10.2-10.37
OSError: [Errno 2] No such file or directory: '/home/roman/tools/python2.7/lib/python2.7/site-packages/tests/fake_packages/FakePlugin.egg/FakePlugin.egg-info'

you might have made the same mistake then I did: install ZopeSkel in your (system-) python installation. Never do this. Mikko explains on his blog why.

So, check your python installation and uninstall/remove the PasteScript.

Patching tzdata and What I Learned From it

The problem

I’ve never paid much thoughts when dealing with dates, times and timezones until I moved to Australia. I should have.


As it happened, if you’re running a server on the east coast of Australia you’re facing three issues:

  • Queensland does not have daylight saving (yay)
  • Tasmania, New South Wales and Victoria have daylight saving
  • but, all of them are in the same timezone.

That means, if you run date on your Linux box you usually get something like that:

 Tue Jun 5 14:04:27 EST 2012

Looks okey, doesn’t it? Not if you start a Zope server:

>>> import DateTime
>>> DateTime.DateTime()
DateTime('2012/06/05 14:06:49.964253 US/Eastern')

Wait, what? I’m in Australia not in the US? This is not New York? It’s sunny Brisbane?

The solution…

Apparently Australians also use AEST instead of EST as an appreviated form to specify the timezone. Ted Percival – facing the same problem – created a patch, which introduces these appreviations in the tzdata package. Just pushing this patch upstream seams not to happen for whatever reasons.
With this patch, I also had to compile a new tzdata rpm for CentOS for Mooball servers. This was a very manual and error prone process. And because I’m personally using Ubuntu I had to catch up with the documentation each time I had to compile and patch a new source rpm.


This lead me to create a package with a Makefile:

  1. It pulls down the source rpm,
  2. applies the supplied patches and
  3. builds a new rpm.

This package registered with Jenkins and you have always an updated RPM for each CentOS release.
Copying the zonefile from e.g. Australia/Brisbane to /etc/localtime resulted in this:

 Tue Jun 5 14:04:27 AEST 2012

and Zope reports:

>>> import DateTime
>>> DateTime.DateTime()
DateTime('2012/06/05 14:06:49.964253 GMT+10')

… is still only a workaround

The problem seems to be application specific. Confirming with the man page on tzname seems to suggest, that applications need to check the timezone and the tzname tuple in order to correctly determine time with daylight saving.

Zope seems to have workarounds added to the DateTime package. Setting a TZ environment variable like:

TZ="AEST" bin/instance fg

will set the correct timezone. Unfortunately it means, that you need to adjust the timezone environment variable on daylight saving (in our case Sydney or Melbourne) and restart your application. Using a patched tzdata helps to avoid restarting applications.

What I learned

So what I now got out of this is:

  • Be more date/time/timezone aware when building applications
  • How to patch source rpms
  • A nice package
    which I can pop into Jenkins which compiles a custom patched rpm.
  • You can adjust the time/date with a simple environment variable, e.g. TZ=”AEST” bin/instance fg
  • Some problems appear as an easy fix, but reveal a more complex situation underneath

Update

Talking to Dylan Jay from Pretaweb, there is more to the environment variable (I should have read the man page of tzname more carefully). Setting the TZ variable to more than just AEST. You can specify beginning and end of daylight saving, eg:

TZ="AEST-10AEDT-11,M10.4.2,M4.4.3" bin/instance fg

The format is extensively documented in the man page of tzname.

handleUidAnnotationEvent from Products.CMFUID throws AttributeError

I stumbled over a weird error during a package setup, which turned out a PEBCAC error. The full traceback:


[...]
File "/home/roman/.buildout/eggs/zope.interface-3.6.3-py2.6-linux-x86_64.egg/zope/interface/adapter.py", line 583, in subscribers
    subscription(*objects)
File "/home/roman/.buildout/eggs/Products.CMFUid-2.2.1-py2.6.egg/Products/CMFUid/UniqueIdAnnotationTool.py", line 86, in handleUidAnnotationEvent
    uid_handler.unregister(ob)
AttributeError: 'NoneType' object has no attribute 'unregister'

I went throught my whole code and type setup unable to find the mistake I’ve made unless I saw the problem. I imported a configuration module for an ArcheTypes content type declaration from a separate package which was not declared in the setup.py of my new package, e.g.:


$ cat my.package.event.py
from separate.package.config import PROJECTNAME
from plone.directives.form import Schema

class IMyType(Schema): ...

If you stumble over this error, make sure your imports are sane.

How *not* to calculate a future date

This mail was in my Inbox today and I was puzzled at first:

When I tried to start Zope it returned an error:

DateError: Invalid date: (2022, 2, 29)

Apparently it only happend on a leap year and exactly on the 29th of
February. Checking the old code base (Python 2.4, Plone 3) revealed this:


DateTime.DateTime(
     DateTime.DateTime().year() + 10,
    DateTime.DateTime().month(),
    DateTime.DateTime().day())

*Doh!*. Python provides a timedelta (Python 2.4 Library reference) module which would be better for calculating future dates.

Update: Apparently others writing similar mistakes 😉 Microsoft’s Azure cloud down and out for 8 hours (via fefe)

Debugging PosKey Errors in ZODB

I ran into a KeyError while packing a ZODB database in production. It lead me to a PosKey error while loading objects which did not exist anymore. For those of you in a similar situation here is what I found very helpful:

Packages I’ve found useful:

  • ZODB3 and it’s provided scripts (fsrefs, fsoids)
  • zc.zodbdgc
  • eye – a ZODB browser

Happy debugging!

Buildout Entry ‘default’ missing?

I recently ran into the following issue with a Plone 3.3.4 buildout:

While:
  Installing.
  Getting section instance.
  Initializing section instance.
  Loading zc.buildout recipe entry plone.recipe.zope2instance:default.

An internal error occured due to a bug in
either zc.buildout or in a recipe being used:
Traceback (most recent call last):
  [...]
  File "/home/roman/.buildout/eggs/
        zc.buildout-1.4.3-py2.4.egg/zc/buildout/buildout.py",
        line 1009, in _install_and_load
    return pkg_resources.load_entry_point(
  File "build/bdist.linux-x86_64/egg/pkg_resources.py",
        line 305, in load_entry_point
  File "build/bdist.linux-x86_64/egg/pkg_resources.py",
        line 2243, in load_entry_point
ImportError: Entry point ('zc.buildout', 'default') not found

I tried to recompile my python 2.4 (because the OS was just newly installed and my lack on a few dev libraries), pin to different buildout versions but couldn’t figure out what was wrong in the finish.

What I ended up doing in the finish, was:

  1. remove my shared buildout directory (e.g. $HOME/.buildout)
  2. rerun a buildout, which created the eggs directory in the buildout directory which seem to work fine
  3. canceled the local buildout run,
  4. recreated the shared buildout directory with my default configuration (e.g. $HOME/.buildout, $HOME/.buildout/default.cfg and $HOME/.buildout/eggs)
  5. rerun the buildout now using the global eggs directory (e.g. $HOME/.buildout/eggs)

Another thing which I’ve noticed, is that zc.buildout does not have a ‘default’ entry point. I checked releases of zc.buildout back to 1.0. I wonder why it seems to be so arbitrarily being loaded that it works sometimes and sometimes not.

Ah … and btw. always a good one: Sometimes removing the .installed.cfg (it’s hidden with a dot in front!) helps as well.