Two Ways to Turbo-Charge tox

No, it’s not (just) run-parallel – let’s cut the local tox runtime by 75%!

Despite being on the record for liking Nox, I still use tox in most of my open-source packages to ensure they work across all supported Python versions, with and without optional dependencies. The brevity of its configuration is a compelling feature for simple cases.

A complete tox run can take a while for various reasons, so finding ways to make it noticeably faster is always great.

The traditional way to speed up tox runs is running it as tox run-parallel (née tox --parallel or just tox -p). And while it’s currently broken in tox 4 for some users (yours truly included), it’s a great feature that Nox is sorely lacking.

But there are more ways, and I’d like to share two of them with you. Both methods don’t make much difference in CIs like GitHub Actions (just like tox run-parallel, mind you!), but they can do wonders for your local development. Which is where I have the least patience, so let’s dive right in!

Step 0: Use tox-uv

Update from 2024-04-11: I will expand this section when I have time, but for now: If you can, install tox-uv along with tox and enjoy much faster creations of virtual environments and much faster package installations without doing anything else.

Wheels on fire

By default, tox will create a virtual environment, build your package’s source distribution (sdist), and install the sdist into the virtual environment – for each environment you define. With env_list = py3{9,10,11}, it will do that three times. With env_list = py3{9,10,11}-{foo,bar}, the number already grows to six¹.

While the virtual environments can be re-used between runs of the same tox environment, the sdist building and installation has to happen for every run. For packages with many environments, that can account for the majority of the runtime of a tox run.

It’s 2023 – wouldn’t it be nicer to build a wheel instead of an sdist, install that into the environments, and run the tests against it? Not only would you have to build it only once – wheels also install much faster than source distributions.

This was always technically possible by building a wheel by hand (e.g., python -m build .) and passing it to tox using the --installpkg option.

With tox 4, tox run --installpkg path/to/pkg.whl still works, but it can also be just two lines of configuration:

[testenv]
package = wheel
wheel_build_env = .pkg

The first line tells tox to build and install wheels instead of source distributions, and the second line tells it to share the same build environment – and thus wheel – across all tox environments.

These two lines reduced the runtime of a full tox run for my service-identity, which has a sub-100ms test run over 47 environments, from 3 minutes to less than 30 seconds.

For a package with a slower test suite (~800ms) and much fewer test environments (17) like my structlog, the difference is much less pronounced: 2 minutes 35 seconds versus 1 minute 47 seconds. But still not too shabby for two lines of configuration!

Three things to keep in mind:

1. Since all Python versions share the same wheel, this only works with universal wheels. But that’s the vast majority, and if you don’t know what that means, you’re most likely OK².

2. In CI, you usually run each Python version in parallel and just a handful of tox environments per Python version. Therefore there is little-to-no performance gain to be made. You can, however, create a wheel as a step in your CI, which is then used by all your Python versions via tox run --installpkg.

3. The wheel build environment that tox uses is not configured using [testenv]. This can be a problem if you need to pass through variables like SETUPTOOLS_SCM_PRETEND_VERSION to the build process. You have to set them in [testenv:.pkg] or [pkgenv]:

   [pkgenv]
   pass_env = SETUPTOOLS_SCM_PRETEND_VERSION
   

Once tox run-parallel works again for everybody, this trick should make it faster than ever!

Parallelization without run-parallel

While tox run-parallel might currently be broken for some of us, it doesn’t mean we can’t parallelize our test suites anyway!

At least you can if you use pytest. You just install the pytest-xdist plugin³ and run pytest -n auto. Shazam! pytest will start as many worker subprocesses as you’ve got CPU cores and distribute the tests among them.

This takes my attrs test suite that is very CPU-heavy – courtesy of property testing – from 10 seconds to 4.

This sounds almost too easy, and indeed, it does come with a bunch of gotchas due to its parallelism:

• Multiple tests running simultaneously virtually rule out any test suites that use permanent databases or similar shared read/write resources.
• -s (= don’t capture output) doesn’t work anymore because otherwise, you’d get overlapping output.
• --pdb (= drop into debugger on errors) silently cancels -n auto.
• Starting those jobs takes a moment – if your test suite is very fast, it may take longer. For example, my doc2dash goes from 100ms to 600ms.

Therefore, I only use it when running all of tox to verify nothing broke before pushing it to GitHub – not while developing (where I always have a shell alias t to run tests). You can achieve that by passing it as default arguments to pytest in your tox.ini:

[testenv]
# ...
commands = coverage run -m pytest {posargs:-n auto}

The ‌{posargs:-n auto} construct allows you to pass -n auto by default but overwrite it using tox run -- --another-pytest-option whenever necessary.

Making Coverage work

Unfortunately, there’s another problem: if you run the configuration from above, you’ll be dismayed to see the following warning:

Coverage.py warning: No data was collected

At the risk of oversimplifying things: Coverage.py needs to be told about subprocesses to measure them by calling:

import coverage
coverage.process_startup()

Since we don’t control the creation of those processes (pytest-xdist does), this is not as straightforward as one would wish. Let’s look at how we can work around this problem.

pytest-cov

The easiest way out is pytest-cov that takes care of this problem entirely, and if it works for you, you can skip the following two sections.

Unfortunately, for me and how I handle coverage information in CI, its mode of operation is a poor match. With the extremely generous help by its maintainer, I got attrs work with it anyway. But I found the result was fighting pytest-cov, relying on semi-documented features. So I’ve decided to solve it myself.

coverage-enable-subprocess

coverage-enable-subprocess gets as close to the ideal of a package that is done as one can.

The package does what its name says: it enables subprocesses in Coverage.py by sneaking in a .pth file into your environment. All you have to do is point the COVERAGE_PROCESS_START env variable to your ‌Coverage.py configuration, and it works like before:

[testenv]
deps =
    coverage[toml]
    coverage-enable-subprocess
    pytest
    pytest-xdist
set_env = COVERAGE_PROCESS_START={toxinidir}/pyproject.toml
commands = coverage run -m pytest {posargs:-n auto}

Looking closer at the source code of coverage-enable-subprocess, it mostly consists of a very custom setup.py using features I don’t have confidence that they won’t get deprecated soon.

And it’s really just adding a trivial coverage_enable_subprocess.pth to site-packages, containing the process initialization code. So I’ve decided to put the damned .pth file there myself.

Manual

Turns out, it’s just one (long) extra line in tox.ini:

[testenv]
deps =
    coverage[toml]
    pytest
    pytest-xdist
set_env = COVERAGE_PROCESS_START={toxinidir}/pyproject.toml

commands_pre = python -c 'import pathlib; pathlib.Path("{env_site_packages_dir}/cov.pth").write_text("import coverage; coverage.process_startup()")'

commands = coverage run -m pytest {posargs:-n auto}

And since this is not tox-specific but a pytest feature, you can do the same thing with Nox:

@nox.session
def tests(session: nox.Session) -> None:
    session.install(".[tests]")

    (
        Path(session.virtualenv.location)
        / "lib"
        / f"python{session.python}"
        / "site-packages"
        / "cov.pth"
    ).write_text("import coverage; coverage.process_startup()")

    session.run(
        "coverage", "run", "-m", "pytest",
        *(session.posargs or ("-n", "auto")),
        env={"COVERAGE_PROCESS_START": "pyproject.toml"}
    )

It’s a bit cumbersome because Nox doesn’t have the equivalent of {env_site_packages_dir}, but it works just as well!

As with the previous tip, this doesn’t do as much in CI, because the GitHub Actions virtual machines only have two cores. But locally, attrs gets from 4.5 to 2.25 minutes which is quite the difference. But again: measure whether your test suite is complex enough to benefit from the parallelization overhead!

Putting it together

I have only one project that has both a lot of environments and a long-running test suite, so let’s see what applying those two methods to attrs gives me on my notebook⁴:

• neither: 257 seconds
• only pytest-xdist: 123 seconds
• both: 69 seconds

Nice!