master.cfg

master.cfg is the config file that Buildbot loads when starting up. As I mentioned above, I’m treating master.cfg as essentially an entry point to the CI server. It’s going to set up some environment stuff, load a bit of config, and then pass over control to the main “app” in the djangobotcfg module.

First, I import some stuff:

import json
import djangobotcfg
from buildbot.manhole import AuthorizedKeysManhole
from unipath import FSPath as Path

Not very exiting, but a couple interesting things from the very beginning:

• I’m using some third-party packages – unipath, here. I’m going to run Buildbot in a venv like a good boy, so I’m going to take full advantage of being able to install stuff. I’ve even got a pip requirements file
• Check out that import djangobotcfg. Yup, I’m putting 99% of the config into a separate module. It’s even got a couple of tests. Just like a real program.

Next, I make sure that things that might change are easy enough to tweak:

SVN = 'http://code.djangoproject.com/svn/django'
BRANCHES = {'trunk': SVN + '/trunk',
            '1.2.X': SVN + '/branches/releases/1.2.X'}

# Load some secrets to pass onto the various bits that need it.
SECRETS = json.load(open(Path('~/master/secrets.json').expand()))

I guess the SVN repo probably change [1], but as we push releases we’ll need to change which branches we test against. We currently build against trunk and the latest bugfix, and I should probably think about adding the security-fix release (1.1.X) to this list, too. I’ll show a bit later on how I make sure that builds only run when code actually gets committed to a particular branch.

Oh, and I also stuffed all the various “secrets” – passwords, mostly – into a secrets.json on the server. That way I don’t have to share all my secrets with the world [2].

In the long run I should probably move more constants up here (or into a JSON file) for easy editing. You’ll see later that there’s more hardcoded stuff – slave names, config info, etc. – that might be best moved into a central location that’s easy to edit. It’s on the ol’ TODO list.

Moving on, check this awesomeness out:

# Bootstrap Django so that authentication against Django's database works.
# Since buildbot sometimes reloads this module we've got to be careful.

from django.conf import settings
if not settings.configured:
    settings.configure(
        INSTALLED_APPS = ['django.contrib.auth'],
        DATABASES = {
            'default': {
                'ENGINE': 'django.db.backends.postgresql_psycopg2',
                'NAME': 'djangoproject',
                'USER': 'djangoproject'
            }
        }
    )

Hey, you got Buildbot in my Django! You got Django in my Buildbot!.

That’s right: I’m going to be using Django in my Buildbot configuration file. If this was just a config file like settings.py I’d think this was sick, but once again: I’m treating this config as a program, and I’m going to take full advantage of every tool I’ve got in my Python toolbox. In particular, I’m going to make Buildbot authenticate against Django’s auth database [3].

Next, it’s time to generate the main parts of the config:

slaves = djangobotcfg.slaves.get_slaves(SECRETS)
status = djangobotcfg.status.get_status(SECRETS)
builders = djangobotcfg.builders.get_builders(BRANCHES, slaves)
schedulers = djangobotcfg.schedulers.get_schedulers(BRANCHES, builders)
changesource = djangobotcfg.changesource.get_change_source(SVN, BRANCHES)

This is calling into my djangobotcfg module, using functions to generate the various parts of config that Buildbot’s going to want. Instead of creating config at module level I’m calling functions to generate that config. This means I can pass in arguments, and it even means I can unit test my config [4].

Finally, we come to the important part. The master.cfg “contract” is that it has to provide a BuildmasterConfig dict. Buildbot imports master.cfg, looks for this BuildmasterConfig dict, and draws all the config from it. So this dict is basically the “table of contents” for the buildbot instance.

In most Buildbot examples I could find – including the out-of-the-box sample config – this dict is built up over the course of many dozens of lines of code. In mine, it’s pretty simple since all the complex parts are buried away in from the djangobotcfg module:

BuildmasterConfig = {
    'slaves': slaves,
    'change_source': changesource,
    'schedulers': schedulers,
    'builders': builders,
    'status': status,
    'slavePortnum': 9989,
    'projectName': 'Django',
    'projectURL': 'http://code.djangoproject.com/',
    'buildbotURL': 'http://buildbot.djangoproject.com/',
    'db_url': 'sqlite:///state.sqlite',
    'manhole': AuthorizedKeysManhole(9990, Path('~/.ssh/authorized_keys').expand()),
}

Let’s go over this dict key by key and talk each part. We’ll look at each piece in more detail in later parts of this series, but for now I’d like to focus on what each key represents and how all the parts fit together.

The first ones are the really important parts:

• "slaves" Slaves are the nodes that actually run builds. This key is a list of slave objects; each corresponds to a specific instance of the Buildbot slave process, usually running on a remote machine.

  In my config these are actually “latent” slaves – nodes that are booted on-demand and shut down as needed – but in many (most?) configs they’re physical machines that are up all the time. I’ll have a whole part about latent slaves towards the end of this series.

• "change_source" The change source is where a build typically starts. Think of the change source as a “collector” for the changes you make to your source code.

  There’s a general-purpose remote trigger for a proprietary Buildbot API, a REST-based remote API, tools for picking up new commits from Github and Bitbucket’s post-commit webhooks, and more.

  I’m using a tool that watches Django’s SVN repository and picks up new commits. It’s fairly simple – just polls SVN every five minutes – but it’s good enough for my purposes.

• "schedulers" Schedulers decide when to filter a build based on some arbitrary criteria, including information passed in from a change source.

  There are tools to trigger builds on a timed basis, which are useful for typical compiled projects where you’d like to provide a daily or nightly build. In the future we might add a process that builds a nightly Django tarball for installation, for example.

  Right now, I’m using schedulers which watch the changes from the change source and trigger new builds based on which branch those commits touched. Thus, if someone commits to the 1.2.X branch we only trigger the tests for that branch, not all branches.

• "builders" Builders are the workhorses of the whole system, and we’ll spend a lot of time looking at them in the next installment. Right now I’ll just say that a builder provides a set of instructions to a slave about how to build a particular “thing”.

  These builds, in our case, are the steps required to run the unit tests on a particular platform. As I went over in part 1, we’ve got a whole bunch of configuration combinations we want to test, so each builder represents one specific combination. For example, a “1.2.X / Python 2.6 / PostgreSQL 8.4 / Linux” builder or a “Trunk / Python 2.7 / MySQL 6.0 / Mac OS X” builder. Each builder needs to know which slaves can successfully run that build, which is why djangobotcfg.builders.get_builders receives the list of slaves as an argument.

• "status" Buildbot separates the actual process of running the tests from the mechanism used to deliver the results. The list of status targets in the``“status”`` key are the bits that report on builds. In my case, I’ve got two: the web interface and an IRC bot that reports when builders transition from success to failure or vice versa.

The next group of keys are just simple configuration:

• "slavePortnum" The port that Buildbot listens for slaves on. You’ll need to know this when setting up slaves, but it’s completely arbitrary.
• "projectName", "projectURL", and "buildbotURL" This are used mostly in the web interface and in other status reporting pieces to provide the name of the project, a link to it, and the main URL of the Buildbot instance.
• "db_url" Recent versions of Buildbot have started moving from storing state in on-disk pickles to a bonafide database. I’m just using SQLite here, but if the database started being a bottleneck (unlikely) I could always switch.
• "manhole" This is actually pretty neat: the “manhole” is actually an interactive Python interpreter embedded in the Buildbot Connect over SSH to the manhole port and you’ll see the familiar >>> prompt! It’s not all that useful for a stable server, but I found it really useful for poking at bits I din’t quite understand. I’m piggy-backing on my SSH keys for access control, which is also sweet.

Build process overview

In practice, here’s how all the above fits together:

1. Somebody commits to Django.
2. The SVN poller – the "change_source" entry – wakes up a few moments later, checks SVN, and notices there’s a new commit. It parses some info out of the commit, and passes the branch name and list of changed files onto the schedulers.
3. Each scheduler in the "schedulers" list gets a crack at at the changeset. In my config there’s one scheduler for each branch, and that scheduler triggers a build if the branch in the changeset matches the scheduler. However, schedulers can really use any technique for deciding whether a change is “interesting”. For example, in the future I’ll probably have a documentation scheduler that triggers a rebuild of the docs whenever a commit is made that involves documentation
4. The scheduler gets to trigger any number of "builders". Again, in my config it’s a simple “which branch?” question, but this could be as arbitrary as needed.
5. Each builder knows which "slaves" it can build on, picks one of those slaves to handle the build. Builders can queue up builds that can’t be handled right now, and I’m using this to make sure that each slave only runs one build at a time (because I’m running on very small instances with only 256 MB of RAM and if they start swapping the tests take for-fucking-ever.)
6. Once a builder has a free slave it generates a set of build steps for that slave. These steps can do pretty much anything as we’ll see in the next part of the series.
7. The slave executes the build steps, reporting success, failure, and progress back to the master.
8. The "status" entries get notified as stuff happens, triggering updates to the web page, notifications over IRC or email, etc.

What’s next?

In the next part I’ll dive into my particular implementation of these various bits – the code for my change sources, schedulers, slaves, builders, status, etc.