TIL: pip-tools Supports pyproject.toml

pip-tools is ready for modern packaging.

I’m on the record for liking pip-tools to pin my production dependencies, because it does one thing well. What I didn’t realize is that it had support for modern pyproject.toml files for well over a year!

That means that you can use pip-compile together with project dependencies defined in pyproject.toml which in turn means that you can use it both for applications as well as for pinning your PyPI package dependencies to keep your CI stable.

And this works with all standard-compliant packaging tools (including Hatch, Flit, and newly even setuptools). The beauty of standardization taking hold in Python packaging land!

Example

Say you want to deploy a pinned Flask application. You define the application and its dependencies in a pyproject.toml:

[build-system]
requires = ["hatchling"]
build-backend = "hatchling.build"

[project]
name = "my_web_app"
classifiers = ["Private :: Do Not Upload"]
version = "0"
dependencies = ["Flask"]

Now you run compile¹:

$ python -m piptools compile \
    -o requirements.txt \
    pyproject.toml

You get:

click==8.1.3
    # via flask
flask==2.2.2
    # via my_web_app (pyproject.toml)
itsdangerous==2.1.2
    # via flask
jinja2==3.1.2
    # via flask
markupsafe==2.1.1
    # via
    #   jinja2
    #   werkzeug
werkzeug==2.2.2
    # via flask

Hashes are possible by adding --generate-hashes, but the output is too unwieldy for a blogpost.

Next, you want to pin your development dependencies, so your tests don’t break randomly. Let’s assume it’s just pytest and amend our pyproject.toml with an dev extra/optional dependency:

[project.optional-dependencies]
dev = ["pytest"]

You run:

$ python -m piptools compile \
    --extra dev \
    -o dev-requirements.txt \
    pyproject.toml

And get:

attrs==22.1.0
    # via pytest
click==8.1.3
    # via flask
flask==2.2.2
    # via my_web_app (pyproject.toml)
iniconfig==1.1.1
    # via pytest
itsdangerous==2.1.2
    # via flask
jinja2==3.1.2
    # via flask
markupsafe==2.1.1
    # via
    #   jinja2
    #   werkzeug
packaging==21.3
    # via pytest
pluggy==1.0.0
    # via pytest
py==1.11.0
    # via pytest
pyparsing==3.0.9
    # via packaging
pytest==7.1.2
    # via my_web_app (pyproject.toml)
tomli==2.0.1
    # via pytest
werkzeug==2.2.2
    # via flask

Now you’ve got a requirements.txt and a dev-requirements.txt generated from one file, where the dependencies live along with the rest of your project’s metadata.

You install them as usual. In production:

$ python -m pip install \
    -r requirements.txt \
    .  # <- the app/pkg itself

While developing:

$ python -m pip install \
    -r requirements.txt \
    -r dev-requirements.txt \
    --editable .  # <- the app/pkg itself

Technically, you only need to pass -r dev-requirements.txt here, since it contains all main dependencies too. I pass them both to ensure I’m getting the same versions of all packages in prod and dev. There’s been cases where an upper pin of a dev dependency lead to different versions of third-party dependencies – that can lead to very confusing bugs and differences in behavior. By passing both, this causes a dependency conflict and pip will fail and expose it to you.

Of course, you can have as many extras and resulting pin files as you want. Each extra/optional dependency is independent from the others, unless you make them depend on each other.

Bonus: How I Keep My Pins Up-to-date

My applications have a Makefile with the following targets:

update-deps:
	pre-commit autoupdate
	python -m pip install --upgrade pip-tools pip wheel
	python -m piptools compile --upgrade --resolver backtracking -o requirements/main.txt pyproject.toml
	python -m piptools compile --extra dev --upgrade --resolver backtracking -o requirements/dev.txt pyproject.toml


init:
	rm -rf .tox
	python -m pip install --upgrade pip wheel
	python -m pip install --upgrade -r requirements/main.txt -r requirements/dev.txt -e .
	python -m pip check

update: update-deps init

.PHONY: update-deps init update

Updating the pin files is a matter of running make update. Installing the project into a fresh virtual environment a matter of make init.