Quitting pip: How we use git submodules to manage internal dependencies that require fast iteration
Philipp Stephan
Everybody has to do it and nobody likes doing it: Dependency management is hard, and it does not help that Python offers you many different approaches.
There are several package managers, like pip, conda or poetry, which offer various ways of distributing packages, e.g. from a git source, local path, or wheels served off a private PyPI server. To deal with different versions, there are a multitude of virtual environment systems like virtualenv, pipenv, and venv. Finding the right workflow and picking the right tools can be a challenge.
After we review the current situation in Python dependency and package management and discuss the advantages and drawbacks of various approaches we tried in the past, we present our current solution to the problem: Using git submodules allows us to iterate quickly on different versions of the same library, keep all the benefits of IDE tooling and ensure that dependency versions are pinned replicably.
We share some insight into the tooling created around the workflow, including automation for adding dependencies easily, CI pipelines to ensure compliance, IDE integration, and synergies with our dockerized tech stack.
