[Python Package Dependency Management]

  • Status: proposed

  • Deciders: TBD

  • Date: 2019-06-18

Context and Problem Statement

We as a team have been struggling with dependency management for a while now. Our current approach (‘manual’ requirements + pip constraints files) is cumbersome and error prone especially in case of dependency package version upgrades. Generally better tool support was seen as a solution to the issue.

Decision Drivers

  • The Process of upgrading a Python package dependency is an error prone, cumbersome and manual process

  • There were multiple occasions of failed dependency upgrades leading to failed CI builds and downstream incompatibilities

Considered Options

Decision Outcome

pip-tools was chosen after a discussion between @hackaugusto, @konradkonrad, @palango and @ulope as it currently seems to be the least disruptive and most well-used tool available. Medium term poetry might become the preferred solution but didn’t appear mature enough currently.

Pros and Cons of the Options

pip-tools

Currently the most mature tool.

  • Pros

    • Small scope, only manages dependencies

    • (Relatively) easy to understand operation model

    • Stable with a long history of being maintained

    • Better dependency solver than pip (which doesn’t have one)

  • Cons

    • No built-in support for dependencies between various requirement types (e.g. prod, dev). Requiring a custom wrapper tool.

    • CLI isn’t very intuitive

poetry

Looks to be a good candidate to switch to in the medium future.

  • Pros

    • Very polished cli

    • Handles the complete package life-cycle including optional venv management

    • Proper dependency solver

  • Cons

    • Still very new with some bugs and some usage types not supported (yet)

    • Very much a departure from the established ‘way of doing things’

    • Dependency resolution can currently be very slow

pipenv

Similar in concept to poetry, yet seems to be not a stable tool to build upon.

  • Pros

    • ?

  • Cons

    • Also a very new tools

    • Many reports of arbitrary breakage with minor upgrades

    • Dependency resolution appears not to be stable