Recursive Optional Dependencies in Python

One of my (slowly evaporating) reasons why I like putting packaging metadata into an executable setup.py is the ability to have optional dependencies that are combinations of others. As of pip 21.2, this is possible without running code.

A package’s optional dependencies (also known as extras) are named sets of dependencies that are installed by putting their names inside square brackets behind the package name. For instance pip install httpx[http2] will install httpx along with optional dependencies that are needed for HTTP/2 support. You can specify more than one extra at once: pip install httpx[http2,cli] will install everything necessary for HTTP/2 and httpx’s CLI interface.

I like to use a combination of optional dependencies in development, by having a dev extra that contains everything needed to run tests, build documentation, and tools that are useful in interactive development, but shouldn’t be present in CI. While running tests I don’t need Sphinx and while I build docs I don’t need pytest. But when I work on the project, I need both along with a nicer debugger or MyPy.

In a setup.py file it looks like this:

extras = {
    "tests": ["pytest"],
    "docs": ["sphinx"],
}

extras["dev"] = extras["tests"] + extras["docs"] + ["pdbpp"]

setup(
    name="my-pkg",
    # ...
    extras_require=extras,
)

pip install -e .[dev] will now install pytest, sphinx, and pdbpp. Thus, my development environment is ready to go. This is particularly useful with public projects, because the effort to set up a local development environment for your contributors shrinks to one line.

This is handy and it stopped me from embracing static configuration using pyproject.toml (PEP 621) for quite a while. Among others, it would mean duplication in my optional dependencies¹.

However, I was also always intrigued by static metadata and modern packaging tools like Hatch or Flit.

Cog: static templating

I tipped my toe into static waters for the first time, when I – along with the Python community – discovered Cog. Cog allows to apply inline templates in static files by hiding the templating logic behind comments.

So I declared my dev-specific dependencies (in this case just pdbpp) followed by a Cog templating block that reads and parses pyproject.toml (itself!) and adds the dependencies from tests and docs:

[project.optional-dependencies]
tests = ["pytest"]
docs = ["sphinx"]

dev = [
    "pdbpp",
    # [[[cog
    # import pathlib, tomli
    # cfg = tomli.loads(pathlib.Path("pyproject.toml").read_text())
    # opt = cfg["project"]["optional-dependencies"]
    # for dep in opt["tests"] + opt["docs"]:
    #     print(f'"{dep}",')
    # ]]]
    "pytest",
    "sphinx",
    # [[[end]]]
]

As you can see, pytest and Sphinx are part of the dev list and whenever I change tests or docs and run Cog, the list between # ]]] and # [[[end]]] gets updated.

To run Cog and to ensure that the file isn’t out of date, I use two tox targets where cogCheck is also run in CI:

[tox]
envlist = cogCheck,cog  # ...

[testenv:cogCheck]
description = "Ensure pyproject.toml is up to date"
skip_install = true
deps = {[testenv:cog]deps}
commands = python -m cogapp --check -P pyproject.toml

[testenv:cog]
description = "Update pyproject.toml's metadata"
skip_install = true
deps =
    cogapp>=3.3.0
    tomli
commands = python -m cogapp -rP pyproject.toml

This is pretty cool and I also use Cog to import and modify my README into PyPI’s long description, so it’s here to stay. However, it’s also kinda clunky, especially for company-internal projects that don’t have a long description.

Which finally brings us to today’s topic!

pip 21.2: recursive dependencies

Python’s packaging progress might be slow but it’s steady. Since pip 21.2 you can refer to your own project in your optional dependencies:

[project]
name = "my-pkg"

[project.optional-dependencies]
tests = ["pytest"]
docs = ["sphinx"]
dev = [
    "my-pkg[tests,docs]",
    "pdbpp",
]

As dev is not used by endusers, the requirement of a bleeding-edge (aka only one year old) pip version is not a big deal.

Here’s an example of how this looks in the wild: structlog’s pyproject.toml.

Bonus tip that might save you some time: the extra names are normalized like package names. An extra foo_bar has to be referred to as foo-bar. Ideally stop using underscores in extra names altogether.

Whither dev?

For someone who’s very outspoken about version pinning, it might be surprising that I use an optional dependency instead of a pin file.

I don’t pin the development dependencies of my open-source packages, because there’s not enough activity to justify the constant commit churn of dependency updates. As it stands now, it’s more practical to fix breaking CI as it happens (which is rare) instead of becoming one of the projects whose majority of commits are dependency updates.

This is my trade-off calculation – yours might be completely different. If your CI breaks regularly due to dependency updates, you should look into pin files and services like Dependabot.

Of course, recursive optional dependencies are useful for more than just this example that I used for illustration.