My Python Development Environment, 2018 Edition

I wrote this post in 2018, more than 6 years ago. It may be very out of date, partially or totally incorrect. I may even no longer agree with this, or might approach things differently if I wrote this post today. I rarely edit posts after writing them, but if I have there'll be a note at the bottom about what I changed and why. If something in this post is actively harmful or dangerous please get in touch and I'll fix it.

This is out of date. For a newer version, see My Python Development Environment, 2020 Edition

For years I’ve noodled around with various setups for a Python development environment, and never really found something I loved – until now.

My setup pieces together pyenv, pipenv, and pipsi. It’s probably a tad more complex that is ideal for most Python users, but for the things I need, it’s perfect.

My Requirements

I do have somewhat specific (maybe unusual?) requirements:

I need to develop against multiple Python versions, including 2.7, various Python 3 versions (3.5 and 3.6, mostly), and PyPy.
I work on many projects simultaneously, each with different sets of dependencies, so some sort of virtual environment or isolation is critical.
I use multiple OSes: macOS at work, and Linux (well, Linux-ish - actually it’s WSL) at home.
I want to avoid using the System-provided Python. On macOS it’s too outdated. On Linux, the system Python is used by the OS itself, so if you hose your Python you can hose your system.
I use a bunch Python-based CLI stuff, like youtube-dl, awscli, doc2dash, etc. I want to be able to install and use them without fussing around with activating environments, but I also don’t want their dependencies to clutter up a global installation.
I deploy stuff mostly to Heroku (personal) and cloud.gov (work). I expect this not to change: I’m spoiled, and never want to manage my own infrastructure again.
Although Docker meets all these requirements, I don’t really like using it. I find it slow, frustrating, and overkill for my purposes.

The Setup

1. pyenv

Why? I need to run multiple Python versions, isolated from the system Python. pyenv makes it easy to install, manage, and switch between those multiple Pythons. As a bonus, pipenv integrates with pyenv and will automatically install missing Python versions if they’re required by a Pipfile.

On my Mac, I installed pyenv from Homebrew (brew install pyenv). On Linux, I used the Github installation technique documented in the installation instructions, which was easy and went smoothly.

Then, I installed some Python versions:

pyenv install 3.6.4
pyenv install 3.5.4
pyenv install 2.7.14
pyenv install pypy3.5-5.10.0

And made sure my default Python was set to the latest and greatest:

pyenv global 3.6.4

2. pipsi

Why? pipsi lets me install Python-based CLI stuff (like youtube-dl, awscli, doc2dash, etc.) without those projects’ dependencies messing up my global Python.

Normally, installing pipsi is easy:

curl https://raw.githubusercontent.com/mitsuhiko/pipsi/master/get-pipsi.py | python

However, at the time I’m writing this (late February 2018), this didn’t work for me; I needed to do some tweaking to get it to work. I followed the workarounds documented in issues #124 and #125. Hopefully this’ll be fixed shortly.

I didn’t do this, but it might be a good idea to install pipsi using the system Python (i.e. run pyenv local system before running the installer). The downside is that this will use a potentially old-ass Python, but the upside is that you won’t break your pipsi install if you delete a pyenv version (like upon an upgrade).

3. pipenv

Why? pipenv handles dependency- and virtual-environment-management in a way that’s very intuitive (to me), and fits perfectly with my desired workflow.

The documentation covers a few different ways to install pipenv. Because I’m already using pipsi, I chose the pipsi-based installation:

pipsi install pew
pipsi install pipenv

What it looks like in use

With this all together, all my use-cases are handled simply:

To start new projects, I just make a directory and type pipenv install ... to start installing my dependencies. Pipenv creates a Pipfile for me, and manages it, and I’m up and running.

To work on existing projects, I clone a repository and either run pipenv install (for projects that already have a Pipfile), or pipenv install -r requirements.txt (which as a side-effect automatically converts a the requirements file to a Pipfile).

If I need to switch Python versions, I run pyenv local <version> in my project directory. I can also add:

[requires]
python_version = "<version>"

to my Pipfile, and pipenv will enforce that version requirement.

When I want to install CLI stuff, I use pipsi:

pipsi install awscli
pipsi install doc2dash
# ... etc

When it comes time to deploy, both Heroku and cloud.gov will read and understand my Pipfile. If I need to deploy to something that doesn’t do Pipfile-based installs, I create a requirements.txt by running pipenv lock --requirements.