Jacob Kaplan-Moss

What to do when PyPI goes down

I wrote this post in 2010, more than 13 years ago. It may be very out of date, partially or totally incorrect. I may even no longer agree with this, or might approach things differently if I wrote this post today. I rarely edit posts after writing them, but if I have there'll be a note at the bottom about what I changed and why. If something in this post is actively harmful or dangerous please get in touch and I'll fix it.

Lately PyPI, the Python package index, has been having some availability issues. When PyPI goes down it really hurts Python developers: we can’t install new packages, bring up new development environments or virtualenvs, or deploy code with depedencies.

Work is ongoing – see PEP 381 – to add much-needed resiliency to PyPI, and the fruits of these labors are starting to become available. In particular, a number of PyPI mirrors are now available: b.pypi.python.org, c.pypi.python.org, and d.pypi.python.org are up and running. Each mirror has fairly up-to-date copies [1] of all the packages and metadata on PyPI.

[1]Exactly how often each mirror updates is up to the maintainer. Each mirror maintains a last-modified file that informs you when the mirror was last updated; it’s at http://<mirror>.pypi.python.org/last-modified.

They’re not totally perfect clones – the public parts of the web interface aren’t on the clones – but they have everything that package installation tools need. This means that when PyPI goes down, you can make a few tweaks, switch to a mirror, and get on with your day.

In the future I expect most Python packaging tools will gain native, transparent support for failing over to these mirrors, but for now you’ll need to manually do so. All of the Python packaging tools support mirrors, each in a slightly different way:

pip

If you’re using a recent pip (0.8.1 or later) use the --use-mirrors flag:

pip install --use-mirrors $PACKAGE

You can also set the PIP_USE_MIRRORS environment variable.

This’ll automatically query the list of mirrors and keep trying until one responds. It can take a bit of time when PyPI’s down as it waits for PyPI to time out, but it’ll work.

For older versions of pip, or if you want to force the use of a particular mirror, use:

pip install -i http://d.pypi.python.org/simple $PACKAGE

If you want to instruct pip to always use the mirror – good if you’ll be doing a lot of installation, or if you’re using pip as part of a bigger automation tool – then put:

[global]
index-url = http://d.pypi.python.org/simple

Into ~/.pip/pip.conf.

virtualenv

Unfortunately, virtualenv uses easy_install to boostrap a new environ, so simply making the pip changes isn’t enough to allow you to create new virtualenvs when PyPI is down.

Instead, you’ll need to make the global change to ~/.pydistutils.cfg desribed. under easy_install, below.

Buildout

To use a mirror with Buildout, add:

[buildout]
index = http://d.pypi.python.org/simple

Into your buildout.cfg.

To use the mirror globally in any Buildout on your system, put those same lines above into ~/.buildout/default.cfg.

If you need to run a Buildout bootstrap.py, note that it currently uses easy_install as part of the bootstrap procedure, so you’ll need to add the lines to ~/.pydistutils.cfg described below.

Like pip, I expect that Buildout will eventually gain native mirror support.

easy_install

Really, you shouldn’t be using easy_install any more – please switch to pip.

If you must, though, you can use a mirror with easy_install:

easy_install -i http://d.pypi.python.org/simple $PACKAGE

To make easy_install use a mirror globally, put:

[easy_install]
index_url = http://d.pypi.python.org/simple

Into ~/.pydistutils.cfg.

Because ``easy_install`` is no longer under active development, I don’t expect ``easy_install`` to ever gain native mirror support. Once again, you should switch to ``pip``.

Correction

I’m informed by PJ Eby, the maintainer of setuptools and easy_install, that those projects aren’t quite dead yet and that he’s got plans to add mirroring support to easy_install.

I apologize for getting my facts wrong about the state of setuptools and easy_install

Still, I suggest that folks switch to pip: it’s got a larger development community, more active forward progress, and it’s a better tool in general.

What’s next?

As I’ve been saying, the next steps here are to add support for automatic failover to a mirror. PEP 381 describes the mirroring protocol and the validation procedure client libraries should take to determine which mirrors exist and which are valid.

I expect that the maintainers of the various packaging tools are working on this problem, but I’m also sure that patches to distribute, pip, virtualenv, and Buildout (and any other tool that requires PyPI to run) would be more than welcome.