Developing Django apps with zc.buildout

Create the Buildout environment

First, create the minimal structure for your app. Make a structure like:

django-shorturls/
    LICENSE
    README
    bootstrap.py
    buildout.cfg
    setup.py
    src/
        shorturls/
            __init__.py

Okay, so this looks like a bunch of crap, but really it’s all either empty files or ones you can copy from elsewhere at this point; we’ll fill it all in later.

Let’s look at what’s here:

LICENSE

If you’re releasing open source software, you need to have a license. Remember that in the absence of a license the default is “All rights reserved.” That is, if there’s no LICENSE, it’s not open source.

I use the BSD license; I always have one floating around I can copy.

README

Code without documentation makes Baby Jesus cry. Please: at least — at the very least — include a README with your contact info and basic usage instructions.

When I’m just getting started I just touch README to remind myself that I need to write one before I’m done.

bootstrap.py

This is Buildout’s bootstrap script. This is a little script that you can include with your package to bootstrap the Buildout environment. You should do so: it saves you from having to install Buildout system-wide, since system-wide packages are the road to hell. This also makes it very easy for others to contribute to your app: they don’t have to know how to use Buildout since they can just run the bootstrap.

Getting this file is easy: wget http://svn.zope.org/*checkout*/zc.buildout/trunk/bootstrap/bootstrap.py

buildout.cfg

Buildout’s config file. We’ll spend some time in here later, but for getting started you only need this to contain:

[buildout]
parts =

setup.py

Eventually this’ll contain the magic distutils incantations so that you can install this app and push it to the Cheeseshop. We’ll get to this in a second.

src/

The convention in the Buildout world appears to be putting the source to modules in a src directory, so might as well follow convention.

We’ll stuff all our code in here, later; now it’s just an empty shorturls module.

Just to prove that the above really is easy, here’s what it looks like in reality:

$ mkdir -p django-shorturls/src/shorturls
$ cd django-shorturls
$ cp ~/Shrapnel/BSD_LICENSE LICENSE
$ touch README
$ wget http://svn.zope.org/*checkout*/zc.buildout/trunk/bootstrap/bootstrap.py
$ cat - > buildout.cfg
[buildout]
parts =
^D
$ touch setup.py
$ touch src/shorturls/__init__.py

One of these days I’ll probably write a shell script to automate this; it’s really quite easy, if a bit repetitive.

Now that our boilerplate is written, we can have Buildout generate the rest of its boilerplate:

$ python bootstrap.py
Creating directory '.../django-shorturls/bin'.
Creating directory '.../django-shorturls/parts'.
Creating directory '.../django-shorturls/eggs'.
Creating directory '.../django-shorturls/develop-eggs'.
Generated script '.../django-shorturls/bin/buildout'.

$ ./bin/buildout
Getting distribution for 'zc.buildout'.
Got zc.buildout 1.2.1.
Upgraded:
  zc.buildout version 1.2.1;
restarting.
Generated script '.../django-shorturls/bin/buildout'.

Now we’ve got the latest version of Buildout, and a nice isolated environment to fool around in. Anything we do inside this django-shorturls directory will be nicely isolated, so we can develop in a little sandbox, protected from anything that might differ in the outside world.

Let’s take a look at what’s here now, with the stuff bootstrap.py and buildout created in bold:

django-shorturls/
    bin/
        buildout
    bootstrap.py
    buildout.cfg
    develop-eggs/
    eggs/
        setuptools-0.6c9-py2.5.egg/
        zc.buildout-1.1.1-py2.5.egg/
        zc.buildout-1.2.1-py2.5.egg/
    parts/
    src/
        shorturls/

What’s all this, then?

/bin/buildout

This is the main entry point to interacting with Buildout. Every time you change your buildout.cfg, you’ll re-run bin/buildout to update the environment to reflect the changes in your config.

Buildout actually keeps track (in a file called .installed.cfg)of what was in your config file the last time you ran buildout and what was installed, so only new or changed stuff gets installed each time.

develop-eggs/

A placeholder for eggs you’re currently developing; we’ll see this used later on.

eggs/

Any eggs that Buildout installs end up here. You can see that in my case I ended up with setuptools (which Buildout builds on top of) as well as two versions of zc.buildout — bootstrap.py installed one, and then buildout automatically detected the availability of a newer version and upgraded it for me.

parts

Certain other bits of Buildout cache, download, or otherwise store things here in parts. Think of this as a staging area and cruft zone.

Make a setup.py

So what we’ll be doing next is starting to develop our shorturls app as a Python package, a.k.a. an “egg.” Buildout requires that the package we’re developing be properly egg-ified, so we need to make a setup.py. A minimal one is fine for now:

from setuptools import setup, find_packages

setup(
    name = "django-shorturls",
    version = "1.0",
    url = 'http://github.com/jacobian/django-shorturls',
    license = 'BSD',
    description = "A short URL handler for Django apps.",
    author = 'Jacob Kaplan-Moss',
    packages = find_packages('src'),
    package_dir = {'': 'src'},
    install_requires = ['setuptools'],
)

Set up Buildout, and a sandbox

Next we need to inform Buildout about the egg we’re developing. While we’re at it, we’ll also generate a python that’s local to this development sandbox (this way we can interact with the code as we develop it without needing to install anything).

Modify your buildout.cfg to read:

[buildout]
parts = python
develop = .
eggs = django-shorturls

[python]
recipe = zc.recipe.egg
interpreter = python
eggs = ${buildout:eggs}

Since this is the first full Buildout recipe, let me pause and explain what’s going on here bit by bit. The important bits to know are:

• buildout.cfg is a INI-style config file, with the added ability to interpolate variables using ${varname} syntax. You can refer to keys defined in other sections as ${section:variable}.
• The file is processed in a sort of waterfall style: work starts with the [buildout] section, and then proceeds through every other section defined in the buildout:parts key. This file has a single part, python.
• Since we’re developing an egg (as opposed to just pulling together outside code), we use develop = . to inform Buildout of that fact. This line basically says that we’re developing a package whose setup.py lives in ., the current directory.
• We want the egg we’re developing to be available to any scripts that run inside this environment, so we set eggs = django-shorturls to get the egg we’re developing installed.
• The next part, [python], we’ll use to generate a Python interpreter local to this Buildout.
• Parts are associated with a “recipe.” Recipes are just other eggs, available from PyPI or elsewhere, that define how to process a part of a Buildout.
• Here, we’ve got recipe = zc.recipe.egg. This recipe is the standard way of installing eggs, creating wrapper scripts and interpreters, and others.
• The interpreter = python bit tells zc.recipe.egg to generate a Python interpreter named python and install it locally to the Buildout.
• We want this interpreter to have our development egg available, so we set eggs = ${buildout:eggs} to copy the list of eggs from the buildout part.

Clear as mud, right?

Now that we’ve changed our buildout.cfg, we re-run bin/buildout to pick up those changes:

$ ./bin/buildout
Develop: '.../django-shorturls/.'
Getting distribution for 'zc.recipe.egg'.
Got zc.recipe.egg 1.2.2.
Installing python.
Generated interpreter '.../django-shorturls/bin/python'.

What happened here was:

• Buildout noted that we’re using a develop egg, ran the local setup.py file, and created a file develop-eggs/django-shorturls.egg-link, a little link file making src/shorturls available to the rest of Buildout.
• We asked for the zc.recipe.egg recipe but didn’t have it already installed, so Buildout fetched and installed it from the Cheeseshop.
• Finally, Buildout generated a bin/python interpreter.

Let’s take a look at what this bin/python interpreter is. We haven’t installed our app anywhere, so using the system Python we can’t see the package:

$ /usr/bin/python
>>> import shorturls
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ImportError: No module named shorturls

However, we can get to our package from this little wrapper:

$ ./bin/python
>>> import shorturls
>>> shorturls
<module 'shorturls' from '.../django-shorturls/src/shorturls/__init__.py'>

This shows off one of the key concepts of Buildout: what happens in Buildout, stays in Buildout. That is, everything that you do inside this isolated environment is only available inside the environment, and doesn’t clutter up your system.

*.pyc
*.pyo
.installed.cfg
bin
develop-eggs
dist
downloads
eggs
parts
src/*.egg-info

Create a test wrapper

Code without tests is broken as designed. Therefore, the next step is to set up an easy way to run tests.

We’ll do this with djangorecipe, a Buildout recipe that automates much of the process of using Django with Buildout. It actually does a lot more than we’ll touch on here, but that’s another show.

Open up src/shorturls/testsettings.py and make a minimal settings file:

DATABASE_ENGINE = 'sqlite3'
DATABASE_NAME = '/tmp/shorturls.db'
INSTALLED_APPS = ['shorturls']
ROOT_URLCONF = ['shorturls.urls']

These four settings are the absolute minimum set required just to get the test suite bootstrapped and running. Note that the DATABASE_NAME given there won’t actually be used — tests under SQLite are run in-memory — but you do need one anyway.

Also, a limitation of the test suite is that it’ll only test apps with a models.py (even an empty one), so:

$ touch src/shorturls/models.py

Next, inform Buildout and djangorecipe of that file and ask it to build a test runner by appending the following to buildout.cfg:

[django]
recipe = djangorecipe
version = 1.0.2
project = shorturls
projectegg = shorturls
settings = testsettings
test = shorturls
eggs = ${buildout:eggs}

Oh, and remember to add django to the list of parts in the buildout:parts setting (so it should now read parts = python django).

Since I’m doing the explain-even-the-obvious thing, let’s go line-by-line:

recipe = djangorecipe

We want to use djangorecipe for this part.

version = 1.0.2

Develop against Django 1.0.2. djangorecipe can take a few options here: we could say version = trunk to develop against the trunk, and a few other options detailed in the djangorecipe docs.

project = shorturls

The name of the Django “project” that the Django installation will use.

The main purpose of djangorecipe is to pull together all the dependancies for a deployed Django app, and in that situation you need a project to contain the settings, root URLconf, and such. The recipe will actually generate this project module for you, which is nice. We don’t want that to happen, however, so…

projectegg = shorturls

This tells djangorecipe that our project module already exists in the package shorturls, so don’t generate a project module.

settings = testsettings

The name (relative to project) of the module containing settings. So between this and project, the DJANGO_SETTINGS_MODULE used by the test runner will be shorturls.testsettings.

test = shorturls

This tells the recipe to generate a test runner for the apps listed. We just have one, shorturls.

eggs = ${buildout:eggs}

Just as for the python part, we want to make the eggs that Buildout knows about available to the generated scripts.

Just as before, since we’ve made changes to the buildout.cfg, we re-run buildout to pick up those changes:

$ ./bin/buildout
Develop: '.../django-shorturls/.'
Installing 'djangorecipe'.
Getting distribution for 'djangorecipe'.
Got djangorecipe 0.17.1.
Installing django.
Downloading Django from: http://www.djangoproject.com/download/1.0.2/tarball/
Generated script '.../django-shorturls/bin/django'.
Generated script '.../django-shorturls/bin/test'.

We’ve got two new scripts here.

bin/django is a manage.py wrapper that bootstraps Django into the Buildout environment. Try, for example, ./bin/django shell to get into a shell already bootstraped both for Buildout and for Django.

bin/test is the one we’re really interested in, though: this is the one that runs the test for the app we’re developing. To make sure everything works, let’s create a simple passing test to double-check that it gets picked up and run correctly. Put something like the following into src/shorturls/tests.py:

from django.test import TestCase

class ShortURLTests(TestCase):
    def test_environment(self):
        """Just make sure everything is set up correctly."""
        self.assert_(True)

And then run the tests:

$ ./bin/tests
Creating test database...
.
------------------------------
Ran 1 test in 0.003s

OK
Destroying test database...

Sweet.

Write some code

Hopefully you already know how to do this part. Just write the app, OK? This isn’t Buildout-specific, but a few hints anyway:

• Designing your application for re-use is a whole field unto itself. James is really the expert here; if you haven’t already, you should check out his talk on building reusable apps (slides, video).
• Please, for the love of $DEITY, do use this test setup. I don’t care if you’re a TDD or a BDD or a DDD kind of guy, but please write tests and run them early and often.
• If it turns out that your app requires some other external package, you can easily make it available by adding it to the list of eggs in the buildout:eggs section. They’ll get installed into the Buildout environment automatically for you.

Upload to the Cheeseshop

Now that we’ve got code, it’s time to share it with the world. To be good citizens, let’s fill out the setup.py with all the extra info a good package should have. Here’s my final setup.py:

import os
from setuptools import setup, find_packages

def read(fname):
    return open(os.path.join(os.path.dirname(__file__), fname)).read()

setup(
    name = "django-shorturls",
    version = "1.0",
    url = 'http://github.com/jacobian/django-shorturls',
    license = 'BSD',
    description = "A short URL (rev=canonical) handler for Django apps.",
    long_description = read('README'),

    author = 'Simon Willison, Jacob Kaplan-Moss',
    author_email = 'jacob@jacobian.org',

    packages = find_packages('src'),
    package_dir = {'': 'src'},

    install_requires = ['setuptools'],

    classifiers = [
        'Development Status :: 4 - Beta',
        'Framework :: Django',
        'Intended Audience :: Developers',
        'License :: OSI Approved :: BSD License',
        'Operating System :: OS Independent',
        'Programming Language :: Python',
        'Topic :: Internet :: WWW/HTTP',
    ]
)

I added a trick to read in the README into the package’s long description; this becomes available on the PyPI detail page. I’ve also added trove classifiers to help people navigate through the Cheeseshop and find my package.

Finally, at long last, we can push our package to the Cheeseshop:

$ ./bin/buildout setup . sdist register upload

This does the three things we need to do to get this package listed:

1. Create a distribution (sdist). This generates a tarball in dist/.
2. Register the package with the Cheeseshop (register). If you’ve not already created a PyPI account this’ll ask you to create one now.
3. Upload your package (upload).

Whiskey

I suggest Bulleit, double, rocks.