Jacob Kaplan-Moss

Python implementation details

I wrote this post in 2009, more than 15 years ago. It may be very out of date, partially or totally incorrect. I may even no longer agree with this, or might approach things differently if I wrote this post today. I rarely edit posts after writing them, but if I have there'll be a note at the bottom about what I changed and why. If something in this post is actively harmful or dangerous please get in touch and I'll fix it.

I’m sitting here at PyCon at the Python Language summit discussing differences between Python versions, and the topic of implementation details came up. The main part of getting Django working on alternate Python VMs was fixing the various assumptions we made about implementation details of Python.

I went back and dug through Django’s ticket tracker for the issues alternate VM issues (most from the Jython developers, some from the PyPy developers) that we had to fix; it’s a pretty instructive list of things you shouldn’t rely on if you’d like your Python code to run on alternate VMs:

  • Using the __builtins__ super-global instead of import __builtin__.
  • Assumptions about the terminal that’s available. Not all terminals will support curses, so anything involving term colors or even isatty is a bit suspect.
  • Assumptions about what’s available in the os module. os.access, os.chmod, os.getpid, may not be available.
  • Relying on features of “old-style” classes, or dynamically creating classes at runtime based on types.ClassType. In general, avoid old-style classes completely if at all possible.
  • Assumptions about function internals, especially im_func.func_code.
  • Assumptions about dict ordering. Remember that dict really is unordered; just because you “usually” get the same order doesn’t mean that you’ve gotten a magic ordered dict.
  • Making code that requires gc.collect.
  • Relying on built-in exception message strings; some (IndexError) differ between VMs.
  • Not paying close attention to type conversion at boundaries (sockets, databases); you might get a boxed type (i.e. a java.lang.String) from the underlying VM where you’re not expecting it.
  • Not thinking carefully about PYTHONPATH, or ignoring things like JYTHONPATH.
  • Not using __import__ properly. Use importlib instead.
  • Assuming the existence or format of .pyc or .pyo objects, or assuming that Python modules must be *.py, or really any assumptions about the names of files (thanks, Leo).