pip-tools is ready for modern packaging.
I’m on the record for liking pip-tools to pin my production dependencies, because it does one thing well. What I didn’t realize is that it had support for modern
pyproject.toml files for well over a year!
That means that you can use
pip-compile together with project dependencies defined in
pyproject.toml which in turn means that you can use it both for applications as well as for pinning your PyPI package dependencies to keep your CI stable.
Note: Instead of the
pip-compile CLI command, I’ll use the
python -m piptools compile incantation for the same reason Brett tells us to use it with pip.
It does the same thing, but ensures you get the correct installation.
Say you want to deploy a pinned Flask application. You define the application and its dependencies in a
[build-system] requires = ["hatchling"] build-backend = "hatchling.build" [project] name = "my_web_app" classifiers = ["Private :: Do Not Upload"] version = "0" dependencies = ["Flask"]
Now you run
$ python -m piptools compile \ -o requirements.txt \ pyproject.toml
click==8.1.3 # via flask flask==2.2.2 # via my_web_app (pyproject.toml) itsdangerous==2.1.2 # via flask jinja2==3.1.2 # via flask markupsafe==2.1.1 # via # jinja2 # werkzeug werkzeug==2.2.2 # via flask
Hashes are possible by adding
--generate-hashes, but the output is too unwieldy for a blogpost.
Next, you want to pin your development dependencies, so your tests don’t break randomly. Let’s assume it’s just pytest and amend our
pyproject.toml with an
dev extra/optional dependency:
[project.optional-dependencies] dev = ["pytest"]
$ python -m piptools compile \ --extra dev \ -o dev-requirements.txt \ pyproject.toml
attrs==22.1.0 # via pytest click==8.1.3 # via flask flask==2.2.2 # via my_web_app (pyproject.toml) iniconfig==1.1.1 # via pytest itsdangerous==2.1.2 # via flask jinja2==3.1.2 # via flask markupsafe==2.1.1 # via # jinja2 # werkzeug packaging==21.3 # via pytest pluggy==1.0.0 # via pytest py==1.11.0 # via pytest pyparsing==3.0.9 # via packaging pytest==7.1.2 # via my_web_app (pyproject.toml) tomli==2.0.1 # via pytest werkzeug==2.2.2 # via flask
Now you’ve got a
requirements.txt and a
dev-requirements.txt generated from one file, where the dependencies live along with the rest of your project’s metadata.
You install them as usual. In production:
$ python -m pip install \ -r requirements.txt \ . # <- the app/pkg itself
$ python -m pip install \ -r requirements.txt \ -r dev-requirements.txt \ --editable . # <- the app/pkg itself
Technically, you only need to pass
-r dev-requirements.txt here, since it contains all main dependencies too. I pass them both to ensure I’m getting the same versions of all packages in prod and dev. There’s been cases where an upper pin of a dev dependency lead to different versions of third-party dependencies – that can lead to very confusing bugs and differences in behavior. By passing both, this causes a dependency conflict and pip will fail and expose it to you.
Of course, you can have as many extras and resulting pin files as you want. Each extra/optional dependency is independent from the others, unless you make them depend on each other.
Bonus: How I Keep My Pins Up-to-date
My applications have a
Makefile with the following targets:
update-deps: pre-commit autoupdate python -m pip install --upgrade pip-tools pip wheel python -m piptools compile --upgrade --resolver backtracking -o requirements/main.txt pyproject.toml python -m piptools compile --extra dev --upgrade --resolver backtracking -o requirements/dev.txt pyproject.toml init: rm -rf .tox python -m pip install --upgrade pip wheel python -m pip install --upgrade -r requirements/main.txt -r requirements/dev.txt -e . python -m pip check update: update-deps init .PHONY: update-deps init update
Updating the pin files is a matter of running
make update. Installing the project into a fresh virtual environment a matter of
Until it’s the default, I recommend passing
--resolver backtrackingto use the great new resolver. ↩︎