Conditional Python Dependencies

Since the inception of wheels that install Python packages without executing arbitrary code, we need a static way to encode conditional dependencies for our packages. Thanks to PEP 508 we do have a blessed way but sadly the prevalence of old setuptools and pip versions make it a minefield to use.

Let’s assume you want to depend on the monotonic package, iff the Python – which your package is being installed on – is older than 3.3 (which added time.monotonic()). Notably this means that the Python version that is building the wheel is different from the Python version that is installing it.

PEP 508

PEP 508 defines python_version as one of the environment markers that are intended for just this purpose along with the syntax:

setup(
    # ...
    install_requires=[
        # ...
        "monotonic ; python_version<'3.3'",
    ]
)

This section has been updated with recent version information.

However, this only works if the build system has setuptools 36.2 (released Jul 14, 2017) or later.

Otherwise the condition is not recorded¹ in the wheel and – in this case – monotonic is always installed.

On the installation side the requirements appear to be a lot looser: pip 1.4 should be fine. Older version will flat-out crash on wheels with conditional dependencies.

For reference:

• Ubuntu Trusty: pip 1.5.4 and setuptools 3.4.4
• Ubuntu Xenial: pip 8.1.1 and setuptools 20.7.0

So neither’s system Python installation is going to give you working wheels but both can install them.

extras_require

Since setuptools 18.0, you can use a very similar syntax that is based on extras_require and that works on older build systems. It is only documented in the wheel docs and it should be avoided nowadays:

setup(
    # ...
    extras_require={
        # ...
        ":python_version<'3.3'": ["monotonic"],
    }
)

When using Pipenv, this approach requires version 11 or later. Previous versions resolve using the Python that is running Pipenv instead using the Python from your virtualenv.

setup.cfg

One common approach used to be to put this information into setup.cfg in a [metadata] section. That approach has been deprecated.

There’s a new approach as of setuptools 36.2.7 based on an [options] section. However the information on its support is rather conflicting as of 2017-12-29 and it seems that the implementation isn’t complete yet.

This article will be updated once I have found a definitive answer.

Fixing sdist

People who run old versions of setuptools are also likely to run an sdist-only system without any wheel support whatsoever. That means that setup.py is invoked on each installation which in turn means we can add it to our requirements dynamically:

INSTALL_REQUIRES = [
    # ...
]
EXTRAS_REQUIRE = {
    # ...
}

if int(setuptools.__version__.split(".", 1)[0]) < 18:
    if sys.version_info[0:2] < (3, 3):
        INSTALL_REQUIRES.append("monotonic")
else:
    EXTRAS_REQUIRE[":python_version<'3.3'"] = ["monotonic"]

setup(
    # ...
    install_requires=INSTALL_REQUIRES,
    extras_require=EXTRAS_REQUIRE,
)

The reason for the barbaric version parsing is that if setuptools is really old and maybe carries some baggage from upstream in its version, pkg_resources.parse_version() and distutils.version.LooseVersion() may fail in weird ways. The first part of versions is usually untouched.

Universal Wheels

Most of the wheels people build are universal: they’re the same for Python 2 and 3. It’s indicated by adding this snippet to your setup.cfg:

[bdist_wheel]
universal = 1

If you use the solution from before, this can become a problem. When building a wheel using a pre-18.0 setuptools, it will either always (if the Python is 3.3 or later) or never (if the Python is older) depend on the extra dependency.

And this is more than an intellectual exercise because modern pips try to wheel and cache any dependency they install. Therefore a modern pip paired with an old setuptools can create cached wheels with broken dependencies.

You have multiple options here:

1. Error on bdist_wheel for old versions of setuptools.
2. Make your wheel not universal and then either upload two versions of it to PyPI…
3. …or override the setting using python setup.py bdist_wheel --universal when you build yourself.

I’ve personally decided for option 1 and added an

assert "bdist_wheel" not in sys.argv

to the < 18 path.

Closing

Do not even think about using this article to disparage the current packaging maintainers. Python packaging made progress in the past years I never thought possible.

But we still have a decade of technical debt to clean up.