Since the inception of wheels that install Python packages without executing arbitrary code, we need a static way to encode conditional dependencies for our packages. Thanks to PEP 508 we do have a blessed way but sadly the prevalence of old setuptools and pip versions make it a minefield to use.
Let’s assume you want to depend on the monotonic
package, iff the Python – which your package is being installed on – is older than 3.3 (which added time.monotonic()
). Notably this means that the Python version that is building the wheel is different from the Python version that is installing it.
PEP 508
PEP 508 defines python_version
as one of the environment markers that are intended for just this purpose along with the syntax:
setup(
# ...
install_requires=[
# ...
"monotonic ; python_version<'3.3'",
]
)
This section has been updated with recent version information.
However, this only works if the build system has setuptools
36.2 (released Jul 14, 2017) or later.
Otherwise the condition is not recorded1 in the wheel and – in this case – monotonic
is always installed.
On the installation side the requirements appear to be a lot looser: pip
1.4 should be fine. Older version will flat-out crash on wheels with conditional dependencies.
For reference:
- Ubuntu Trusty:
pip
1.5.4 andsetuptools
3.4.4 - Ubuntu Xenial:
pip
8.1.1 andsetuptools
20.7.0
So neither’s system Python installation is going to give you working wheels but both can install them.
extras_require
Since setuptools
18.0, you can use a very similar syntax that is based on extras_require
and that works on older build systems. It is only documented in the wheel docs and it should be avoided nowadays:
setup(
# ...
extras_require={
# ...
":python_version<'3.3'": ["monotonic"],
}
)
When using Pipenv, this approach requires version 11 or later. Previous versions resolve using the Python that is running Pipenv instead using the Python from your virtualenv.
setup.cfg
One common approach used to be to put this information into setup.cfg in a [metadata]
section. That approach has been deprecated.
There’s a new approach as of setuptools
36.2.7 based on an [options]
section. However the information on its support is rather conflicting as of 2017-12-29 and it seems that the implementation isn’t complete yet.
This article will be updated once I have found a definitive answer.
Fixing sdist
People who run old versions of setuptools
are also likely to run an sdist-only system without any wheel support whatsoever. That means that setup.py
is invoked on each installation which in turn means we can add it to our requirements dynamically:
INSTALL_REQUIRES = [
# ...
]
EXTRAS_REQUIRE = {
# ...
}
if int(setuptools.__version__.split(".", 1)[0]) < 18:
if sys.version_info[0:2] < (3, 3):
INSTALL_REQUIRES.append("monotonic")
else:
EXTRAS_REQUIRE[":python_version<'3.3'"] = ["monotonic"]
setup(
# ...
install_requires=INSTALL_REQUIRES,
extras_require=EXTRAS_REQUIRE,
)
The reason for the barbaric version parsing is that if setuptools
is really old and maybe carries some baggage from upstream in its version, pkg_resources.parse_version()
and distutils.version.LooseVersion()
may fail in weird ways. The first part of versions is usually untouched.
Universal Wheels
Most of the wheels people build are universal: they’re the same for Python 2 and 3. It’s indicated by adding this snippet to your setup.cfg
:
[bdist_wheel]
universal = 1
If you use the solution from before, this can become a problem. When building a wheel using a pre-18.0 setuptools
, it will either always (if the Python is 3.3 or later) or never (if the Python is older) depend on the extra dependency.
And this is more than an intellectual exercise because modern pip
s try to wheel and cache any dependency they install. Therefore a modern pip
paired with an old setuptools
can create cached wheels with broken dependencies.
You have multiple options here:
- Error on
bdist_wheel
for old versions ofsetuptools
. - Make your wheel not universal and then either upload two versions of it to PyPI…
- …or override the setting using
python setup.py bdist_wheel --universal
when you build yourself.
I’ve personally decided for option 1 and added an
assert "bdist_wheel" not in sys.argv
to the < 18
path.
Closing
Do not even think about using this article to disparage the current packaging maintainers. Python packaging made progress in the past years I never thought possible.
But we still have a decade of technical debt to clean up.
setuptools
36.1 appears to be recording the condition but for some reason it wasn’t picked up on installation in my experiments. ↩︎