(Locally) Testing ansible deployments

I’ve always felt my playbooks undertested. I know about a possible solution of spinning up new OpenStack instances with the ansible nova module, but felt it to be too complex as a good idea to implement. Now I’ve found a quicker way to test your playbooks by using Docker.

In principal, all my test does is:

1. create a docker container
2. create a copy of the current ansible playbook in a temporary directory and mount it as a volume
3. inside the docker container, run the playbook

This is obviously not perfect, since:

• running a playbook locally vs connecting via ssh can be a different beast to test
• can become resource intensive if you want to test different scenarios represented as docker images.

There is possibly more, but for myself in small it is a workable solution so far.

Find the code on github if you’d like to have a look. Improvements welcome!

 

Ansible Variables all of a Sudden Go Missing?

I’ve written a playbook which deploys a working development environment for some of our internal systems. I’ve tested it with various versions of RHEL. Yet when I ran it against a fresh install of Fedora it failed:

fatal: [192.168.1.233] => {'msg': "One or more undefined variables: 'ansible_lsb' is undefined", 'failed': True}

It turned out, that ansible gets it’s facts through different programs on the remote machine. If some of these programs are not available (in this instance it was lsb_release) the variables are not populated resulting in this error.

So check if all variables you access are indeed available with:

$ ansible -m setup <yourhost>

Common docker pitfalls

I’ve ran into a few problems with docker I’d like to document myself and how to solve them.

Overwriting an entrypoint

If you’ve configured a script as an entrypoint which fails, you can run the docker image with a shell in order to fiddle with the script (instead of continously rebuilding the image):

#--entrypoint (provides a new entry point which is the nominated shell)
docker run -i --entrypoint='/bin/bash'  -t f5d4a4d6a8eb

Possible errors you face otherwise are these:

/bin/bash: /bin/bash: cannot execute binary file

Weird errors when building the image

I’ve ran into this a few times. Errors like:

Error in PREIN scriptlet in rpm package libvirt-daemon-0.9.11.4-3.fc17.x86_64
or
useradd: failure while writing changes to /etc/passwd

If you’ve set SELinux to enforcing, you may want to temporarily disable SELinux for just building the image. Don’t disable SELinux permanently.

Old (base) image

Check if your base image has changed (e.g. docker images) and pull it again (docker pull <image>)

Deploying Plone 2 with a ZEO Server

At Mooball, there are still instances running with a CMFPlone v. 2. Very, very old. Back in the days when Plone v2 was shipped, Python 2.3 was hip and zc.buildout was never heard of.
One motivation I have to deploy with a ZEO is the possibility to pack your database from a simple cron script (which you configure in your buildout too). If you’re still one of them who has to deal with these old instances, here is what you do:

Use Python 2.3

I’ve always compiled my own python 2.3 to avoid any side effects introduced by python installations from the distribution. Version 1.2.1 of zc.buildout works fine for me, the only missing thing is the subprocess module introduced in python 2.4. To get buildout running, I’ve copied subprocess.py from python 2.4 into the python 2.3 installation. After that is done, run the bootstrap with (the -v pins the zc.buildout version to 1.2.1):

/path/to/python2.3/bin/python bootstrap.py -v1.2.1

Run with zc.buildout

I’ve created a sample buildout.cfg and put it up on gist. The important part is the ‘fixup’ which simply comments out 3 keys in the generated config files, which are not supported by the Zope/ZEO version. Depending how much time you have available, play with the options to pin eggs instead of dropping the whole CMFPlone tarball in the products directory. Running the buildout from the products directory is the quickest way tho.

ZODB/ZEO Error Masking Wrong Filesystem Permissions

One of the clients staging instances showed quite a few tracebacks of that type:

Traceback (innermost last):
  ...
  Module zope.contentprovider.tales, line 77, in __call__
  Module zope.viewlet.manager, line 112, in update
  Module zope.viewlet.manager, line 118, in _updateViewlets
  Module my.product.viewlets.industry, line 29, in update
  Module Products.ZCatalog.Lazy, line 190, in __getitem__
  Module Products.ZCatalog.Catalog, line 121, in __getitem__
  Module ZODB.Connection, line 860, in setstate
  Module ZODB.Connection, line 901, in _setstate
  Module ZEO.ClientStorage, line 815, in load
  Module ZEO.cache, line 143, in call
  Module ZEO.cache, line 494, in load
  Module ZODB.fsIndex, line 125, in get
TypeError: mybrains.__cmp__(x,y) requires y to be a 'mybrains', not a 'NoneType'

I’m getting to the point, since it had something to do with the deployment. Some packages were deployed as develop eggs on the instance. Only a few files were created with the wrong filesystem permissions, so neither Zope nor ZEO were able to access them. After changing the permissions the errors were gone. So be sure to check all filesystem permissions and if the processes have access to them. If that doesn’t solve it, be sure to also check if not your database is corrupt.