Thursday, September 03, 2015

New/alternate home

I'm fickle with blogging services. I've written some new posts over at the confusingly named I may come back here (to Blogger), I may stay there (Logdown), or I may move on again, but I'll try to leave some breadcrumbs as I go.

Wednesday, July 29, 2015

WTF is Dart?

I wrote an opinion piece about Dart for the Norfolk Tech Journal several months ago. It is fairly opinionated, I'm more mellow about JavaScript nowadays, and it is also a little out of date; for example, Dart will not now become part of Chrome. It's here as a personal historical artefact as much as anything.

Read it here:

WTF is Dart?

for Norfolk Tech Journal, 2015-01-12

Internally, Google is reputed to use Java and C++ almost exclusively, with a little Python, but they’ve been developing the Go and Dart languages in the last few years. Go is interesting and revolutionary like celery is spicy yet is burning brightly in the tech community, with lots of high profile adopters. Dart, whilst not revolutionary either, does do useful and novel things, and I think it deserves a little more attention than it gets.

Douglas Crockford’s book Javascript: The Good Parts is slim, yet boasts two enlightening anti-­appendices — Awful Parts and Bad Parts — and the rest of the book is not massively compelling either. However, JavaScript does have two good parts: first­-class functions, and ubiquity.

Node.js extends this ubiquity to the server­-side, but ultimately you’re still writing JavaScript — well, ECMAScript 5 — which brings all­-too-­familiar problems with it.

Now, some people love JavaScript, and I don’t mean to knock them. The development experience can be so quick to iterate that it feels massively productive, and browser competition has produced several good implementations. Those who immerse themselves in JavaScript can do wondrous things, but it can be a very potholed road for those of us who don’t do it full-­time.

I’ve personally found that building larger applications is hard work with JavaScript, and I’m not alone. As an application grows you can find yourself getting uncomfortably intimate with the libraries and frameworks you’re using. Boilerplate and conventions can become the dominant forces in keeping the teetering stack up in the air, and the pleasures of programming slowly file out stage left.

This is where Dart makes sense. It rides the ubiquity of JavaScript by compiling to JavaScript, and provides the convenience of Node.js by being available both client­-side and server-­side. However, it bolts this onto a much cleaner language, with a modest but well­-structured standard library. The language itself has a greater range of data types than JavaScript, functions are (still) first-­class objects, exception handling is richer, there are conveniences like typedefs, reflection, and a library/module system, language support for asynchronous IO, and even generics.

The standard set of tools from includes a decent IDE, profiler and debugger, documentation generator, and Dartium, which is a version of Chromium with native Dart support. This will eventually make it into stable Chrome builds, but it’s good for development right now. Production deployment via compilation to JavaScript is still the way to go on the client, but on the server you can run your code in the Dart VM.

Dart is thoughtfully constructed and well executed. There are the tools you need as a developer, including a unit test library, and command-­line access to the compiler, static analyser, package manager, and others, meaning you’re not tied to Dart’s IDE either. It interoperates with JavaScript on the client, there are ports of Polymer and AngularJS to get started with, or you can interact with JavaScript directly via the dart:js library from the standard library, and code packages can be used for deployment and sharing. If it sounds as if Dart is a mature and well­-designed language that fits nicely into the node.js niche, only without legacy baggage... that’s because it is.

Tuesday, May 12, 2015

South, South 2, and Django Migrations

A couple of months ago we on the MAAS team found ourselves in a bit of a pickle: we needed to be able to support a product targeted at both Django <1.7 and Django ≥1.7 with database migrations. This is a problem because South is replaced by Django's own migration support in 1.7, and there are differences.

I emailed Andrew Godwin to ask his advice. He's the author of South 2 and so apparently knows his stuff, but we also wondered if South 2 might be a way out of our mess. His reply confirmed him as knowledgable, kind, and helpful. Although he did not bless South 2 as our silver bullet, he did have some other useful advice instead.

I promised I would document our correspondence where others might learn from it, and this is it, somewhat overdue. I've edited it slightly for clarity.

Thanks Andrew!

Hi Andrew,

I found your south2 repository on GitHub today. It looks like you've not touched it in a while, but I wondered if I could ask you a few questions about it anyway? There's a lot of context but it boils down to two-ish questions:

  1. What would you recommend for transitioning a packaged product (i.e. one which we don't provide as a service) from South-based migrations to Django ≥1.7 migrations?

    As a general answer, I suggest the method described in the Django docs, which is to move the South migrations to a south_migrations directory and generate new initial Django ones. As long as your users have South 1.0 or higher, that'll keep both versions running during a transition, and Django's automatic application of initial migrations makes things a lot easier. I don't recommend that you try and support both migration sets at the same time; make 1.7 or higher a hard dependency for a release. This obviously is a bit different for the case below, which I answered down there.

  2. How much work would be required to get south2 working?

    It was abandoned with good reason - it's around another two months of work to get it working remotely reliably, and I'm not sure it could be done at all without much more of a rewrite rather than the current source translation approach. I didn't abandon the idea lightly, but alas it just wasn't proving very stable.

We're in a tricky situation:

  • We have an application, MAAS, that we ship as a package in Ubuntu, i.e. end-users install it. It uses PostgreSQL.
  • It's supported in Ubuntu 14.04 (Trusty) and will be supported until April 2019. Trusty ships with Django 1.6, and this won't change (only security fixes and fixes for very serious bugs are back-ported).
  • Django 1.7 is now available in the development version of Ubuntu (Vivid).
  • Django 1.7 or later will be in the next LTS (Long Term Support) version of Ubuntu, out next year. (Trusty is the most recent LTS release.)
  • We have been using South for several years.
  • To support MAAS in Trusty we may need to back-port migrations from trunk. Once we base trunk on Django ≥1.7 we can't back-port directly; we'd need to recreate any migrations with South.
  • However, we also need a seamless upgrade path for users on Trusty when they upgrade to the next LTS release, where they can skip right over three intermediate releases of Ubuntu.
  • Between Trusty and the next LTS (hereafter just "Next"), the upgrade path might look like (where mXXX = "migration XXX"):
    Trusty -- m134 -- m135 -- m136 -- m137  (then EOL)
              \       \       \       \
       Next -- m0 ---- m1 ---- m2 ---- m3 ---- m4 ---- ...

    In other words, Trusty and the next LTS share a common ancestor in South migration 134; the Django ≥1.7 migration baseline is derived at that point.

    At any point after that a user could choose to upgrade to the next LTS. If they upgrade from an installation that's got m136, we could map that over to m2 in the new migrations model, tell Django to fake-apply m0, m1, and m2, then proceed from there.

  • In truth, a user could choose to upgrade from Trusty to Next before having applied m134 because users can choose to follow only security fixes, and not updates. (They can choose to follow nothing at all, but that's getting into a very grey area w.r.t. support.)

    In this situation we'd want to apply all remaining South migrations up to at least m134 before switching over to the new Django migrations model.

    On the other hand, there may be a way to prevent a Trusty → Next upgrade based on a precondition, e.g. "m134 or greater is needed", but I don't currently know how that would be implemented.

  • There's a risk of South migrations not matching up to Django ≥1.7 migrations. That would most likely be an issue with our process, but it could be a software issue too.
  • With a variety of automated testing we can mitigate a lot of the process risk, and catch software issues early.
  • However, that all adds up to quite a lot of work.
  • Another option entirely would be for us to invest time into south2 and switch everything over to Django ≥1.7 migrations. That sounds like it would be a lot simpler, and thus carry a lot less risk.
  • The thing I don't know, which I hope you can answer, is how much work might it be to get south2 to a point where this would be possible? What would the ongoing maintenance look like?
  • What would you recommend?

    There's no clean solution, sorry. I'd document having to apply the most recent migrations before switching (and perhaps have a code entry on startup in the 1.7 dependent version that checks the south_migrations table directly and hard fails if you didn't), then have people clean switch over to the latest release.

    Can I ask why you won't just ship a newer version of Django with the newer releases of MAAS, even on Trusty? I know OS packaging is a tough thing to get around, but trying to backport migrations to work on South and older releases is only going to bring you pain (South is much more limited than Django migrations, and you might have to do a lot of manual workarounds).

    South2 isn't going to work - don't go down that path, I abandoned it for good reason, I'm not even sure the automated source translation approach is possible and a rewrite would take months. You're better off somehow shipping 1.7 bundled or as some kind of special dependency.