Wednesday, November 20, 2013

Preparing for Python 3 in MAAS

Something we've done in MAAS — which is Python 2 only so far — is to put:

from __future__ import (

__metaclass__ = type

str = None

at the top of every source file. We knew that we would port MAAS to Python 3 at some point, and we hoped that doing this would help that effort. We'll find out if that's the case soon enough: one of our goals for this cycle is to port MAAS to Python 3.

The str = None line forces us to use the bytes synonym, and thus think more about what we're actually doing, but it doesn't save us from implicit conversions.

In places like data ingress and egress points we also assert unicode or byte strings, as appropriate, to try and catch ourselves slipping up. We also encode and decode at these points, with the goal of using only unicode strings internally for textual data.

We never coerce between unicode and byte strings either, because that involves implicit recoding; we always encode or decode explicitly.

In maas-test — which is a newer and smaller codebase than MAAS — we drop the str = None line, but use tox to test on both Python 2.7 and 3.3. Unfortunately we've recently had to use a Python-2-only dependency and have had to disable 3.3 tests until it's ported.

maas-test started the same as maas: a Python 2 codebase with the same prologue in every source file. It's anecdotal, but it turned out that few changes were needed to get it working with Python 3.3, and I think that's in part because of the hoops we forced ourselves to jump through. We used six to bridge some gaps (text_type in a few places, PY3 in too), but nothing invasive. My fingers are crossed that this experience is repeated with MAAS.

Thursday, November 07, 2013

Even Google can't write shell scripts

Google's Shell Style Guide — via Hacker News — doesn't mention using set -e or set -u, and neither does Gentoo's guide, which seriously undermines them both. They both have sensible advice, but those two settings are the two most important things to include when scripting with Bash. Bash doesn't even become a scripting language until those are set; without, scripts are just interactive transcripts without the interactive part, i.e. a human to stop execution when there are errors or unexpected behaviour.

My advice, before you read either of these guides, is to put the following at the top of every script you write, before you start: set -eu. When working with a team put in the long version, with comments:

# Exit immediately if a command exits with a non-zero status.
set -o errexit
# Treat unset variables as an error when substituting.
set -o nounset

If you know me, you'll know that that's not really my advice. My really real advice is: don't write scripts with Bash, or most other shells. Learn and use Python instead, for example. Sure, shell scripts are convenient, but you really need 5+ years of writing shell scripts to understand enough of the many insidious ways that they'll break on you. By which time I truly hope you will have learned that you shouldn't use them except as short-lived conveniences. Even then I'm not sure.

Fwiw, neither Google nor Gentoo mention testing either.

Saturday, November 02, 2013


python-liblockfile is a wrapper around liblockfile. It uses ctypes but unfortunately has to hard-code some constants from lockfile.h, though I'll bet they're fairly unlikely to change any time soon. It has tests!

Install it from PyPI: pip install python-liblockfile

There is a low-level API that mirrors liblockfile: lockfile_create, lockfile_remove, lockfile_touch, and lockfile_check. However, for all of these except lockfile_check it uses ctypes' errcheck feature to raise exceptions when the return is non-zero:

Recipient name too long.
Error creating tmp lockfile.
Can't write pid into tmp lockfile.
Failed after max. number of attempts.
Unknown error; check errno.
Cannot set mandatory lock on tempfile.

There's also a more Pythonic, higher-level API: Lock and LockHolder. The former encapsulates the lower-level API and also works as a context manager. The latter is meant to be used as a context manager too, and it works with the time-out feature of liblockfile by spawning a daemon thread to periodically touch a lock. Here's how you might use both together:

from liblockfile import Lock, LockHolder
with Lock("filename-to-lock") as lock, LockHolder(lock):
    # ... do things that require the lock to be held.

Lock has two other constructors: WithPID and WithPPID. See the docs for L_PID in lockfile_create(3); L_PPID can be extrapolated from that.

If you don't need compatibility with liblockfile then you'll probably be better off using a pure-Python locking library — lockfile for example — to avoid native dependencies. This library might still be interesting as a small piece of example code for using ctypes; it's by no means exhaustive, and might get things wrong, but I am open to criticism and patches.

Friday, November 01, 2013

Protocol buffers for logging?

I'd always thought of Protocol Buffers as an on-the-wire structured message format only, but using them for logs seems head-smackingly useful. I'm a little ashamed I didn't think of it before, especially because their description clearly states:

Google uses Protocol Buffers for almost all of its internal RPC protocols and file formats.

Adam D'Angelo's answer on Quora was what finally made me twig.

There are probably limitations, many of them surely similar to the criticisms levelled at systemd's journal, but for logs meant to be re-consumed by machines — before presentation to a human perhaps — I think they're interesting to consider. For example, MAAS's logging is due some love in the coming months as part of our efforts to make it much easier to debug.

fsync the parent directory when fsyncing a new file

You should call fsync() on the parent directory—that is, the directory containing your file—when you need to ensure that a newly-created file is fully synchronized back to disk. In that situation, you would call fsync() on both your file and its parent directory.
Robert Love in response to When should you fsync the containing directory in addition to the file itself? on Quora