What to do when PyPI goes down
Lately PyPI, the Python package index, has been having some availability issues. When PyPI goes down it really hurts Python developers: we can’t install new packages, bring up new development environments or virtualenvs, or deploy code with depedencies.
Work is ongoing – see
PEP 381 – to add much-needed
resiliency to PyPI, and the fruits of these labors are starting to
become available. In particular, a number of PyPI mirrors are now
available: b.pypi.python.org, c.pypi.python.org, and
d.pypi.python.org are up and running. Each mirror has fairly
up-to-date copies [1] of all the packages and metadata on
PyPI.
| [1] | Exactly how often each mirror updates is up to the maintainer. Each mirror maintains a last-modified file that informs you when the mirror was last updated; it’s at http://<mirror>.pypi.python.org/last-modified. |
They’re not totally perfect clones – the public parts of the web interface aren’t on the clones – but they have everything that package installation tools need. This means that when PyPI goes down, you can make a few tweaks, switch to a mirror, and get on with your day.
In the future I expect most Python packaging tools will gain native, transparent support for failing over to these mirrors, but for now you’ll need to manually do so. All of the Python packaging tools support mirrors, each in a slightly different way:
pip
If you’re using a recent pip (0.8.1 or later) use the --use-mirrors
flag:
pip install --use-mirrors $PACKAGE
You can also set the PIP_USE_MIRRORS environment variable.
This’ll automatically query the list of mirrors and keep trying until one responds. It can take a bit of time when PyPI’s down as it waits for PyPI to time out, but it’ll work.
For older versions of pip, or if you want to force the use of a
particular mirror, use:
pip install -i http://d.pypi.python.org/simple $PACKAGE
If you want to instruct pip to always use the mirror – good if
you’ll be doing a lot of installation, or if you’re using pip as part
of a bigger automation tool – then put:
[global]
index-url = http://d.pypi.python.org/simple
Into ~/.pip/pip.conf.
virtualenv
Unfortunately, virtualenv uses easy_install to boostrap a new
environ, so simply making the pip changes isn’t enough to allow you to
create new virtualenvs when PyPI is down.
Instead, you’ll need to make the global change to ~/.pydistutils.cfg
desribed. under easy_install, below.
Buildout
To use a mirror with Buildout, add:
[buildout]
index = http://d.pypi.python.org/simple
Into your buildout.cfg.
To use the mirror globally in any Buildout on your system, put those
same lines above into ~/.buildout/default.cfg.
If you need to run a Buildout bootstrap.py, note that it currently
uses easy_install as part of the bootstrap procedure, so you’ll need
to add the lines to ~/.pydistutils.cfg described below.
Like pip, I expect that Buildout will eventually gain native mirror
support.
easy_install
Really, you shouldn’t be using easy_install any more – please
switch to pip.
If you must, though, you can use a mirror with easy_install:
easy_install -i http://d.pypi.python.org/simple $PACKAGE
To make easy_install use a mirror globally, put:
[easy_install]
index_url = http://d.pypi.python.org/simple
Into ~/.pydistutils.cfg.
Because ``easy_install`` is no longer under active development, I
don’t expect ``easy_install`` to ever gain native mirror
support. Once again, you should switch to ``pip``.
Correction
I’m informed by PJ Eby, the maintainer of setuptools and
easy_install, that those projects aren’t quite dead yet and that he’s
got plans to add mirroring support to easy_install.
I apologize for getting my facts wrong about the state of setuptools
and easy_install
Still, I suggest that folks switch to pip: it’s got a larger
development community, more active forward progress, and it’s a better
tool in general.
What’s next?
As I’ve been saying, the next steps here are to add support for automatic failover to a mirror. PEP 381 describes the mirroring protocol and the validation procedure client libraries should take to determine which mirrors exist and which are valid.
I expect that the maintainers of the various packaging tools are working on this problem, but I’m also sure that patches to distribute, pip, virtualenv, and Buildout (and any other tool that requires PyPI to run) would be more than welcome.