Conditional Python Dependencies

Since the inception of wheels that install Python packages without executing arbitrary code, we need a static way to encode conditional dependencies for our packages. Thanks to PEP 508 we do have a blessed way but sadly the prevalence of old setuptools versions makes it a minefield to use.

Let’s assume you want to depend on the monotonic package, if the Python – which your package is being installed on – is older than 3.3 (which added time.monotonic()). This most notably means that the Python version that is building the wheel is different from the Python version that is installing it.

PEP 508

How hard can that be, since PEP 508 defines python_version as one of the environment markers that are intended for just this purpose along with the syntax:

setup(
    # ...
    install_requires=[
        # ...
        "monotonic ; python_version<'3.3'",
    ]
)

Sadly this – as of setuptools 21 – does silently nothing.

Working Approach

Since setuptools 18.0, you can use a very similar syntax that is based on extras_require and that actually works (but is only documented in the wheel docs):

setup(
    # ...
    extras_require={
        # ...
        ":python_version<'3.3'": ["monotonic"],
    }
)

However this explodes on any version of setuptools older than 18.0. For internal packages, it doesn’t matter and you should go for it. But if you put something on PyPI and don’t want to spend your time closing bogus bug reports, you should read on.

setup.cfg & [metadata]

One common approach used to be to put this information into setup.cfg. That prevented the dependencies from being added but at least nothing broke. However according to the wheel docs, this is deprecated.

Fixing sdist

People who run old versions of setuptools are also likely to run an sdist-only system without any wheel support whatsoever. That means that setup.py is invoked on each installation which in turn means we can add it to our requirements dynamically:

INSTALL_REQUIRES = [
    # ...
]
EXTRAS_REQUIRE = {
    # ...
}

if int(setuptools.__version__.split(".", 1)[0]) < 18:
    if sys.version_info[0:2] < (3, 3):
        INSTALL_REQUIRES.append("monotonic")
else:
    EXTRAS_REQUIRE[":python_version<'3.3'"] = ["monotonic"]

setup(
    # ...
    install_requires=INSTALL_REQUIRES,
    extras_require=EXTRAS_REQUIRE,
)

The reason for the barbaric version parsing is that if setuptools is really old and maybe carries some baggage from upstream in its version, pkg_resources.parse_version() and distutils.version.LooseVersion() may fail in weird ways. The first part of versions is usually untouched.

Universal Wheels

Most of the wheels people build are universal: they’re the same for Python 2 and 3. It’s indicated by adding this snippet to your setup.cfg:

[bdist_wheel]
universal = 1

If you use the solution from before, this can be a problem. When building a wheel using a pre-18.0 setuptools, it will either always (if the Python is 3.3 or later) or never (if the Python is older) depend on the extra dependency.

And this is more than an intellectual exercise because modern pips try to wheel and cache any dependency they install. Therefore a modern pip paired with an old setuptools can create cached wheels with broken dependencies.

You have multiple options here:

  1. Error on bdist_wheel for old versions of setuptools.
  2. Make your wheel not universal and then either upload two versions of it to PyPI…
  3. …or override the setting using python setup.py bdist_wheel --universal when you build yourself.

I’ve personally decided for option 1 and added an

assert "bdist_wheel" not in sys.argv

to the < 18 path.

Closing

Do not even think about using this article to disparage the current packaging maintainers. Python packaging made progress in the past years I never thought possible.

But we still have a decade of technical debt to clean up.

Acknowlegements

Thanks to Donald Stufft for being the only person who understands Python packaging and has the patience to explain. And the amazing Frinkiac that made it possible to express my feelings.