Surprising Consequences of macOS’s Environment Variable Sanitization

Or: Why does DYLD_LIBRARY_PATH keep disappearing!?

One unfortunate fact of my life is that I have to deal with an obscure database whose macOS drivers require the addition of a directory to DYLD_LIBRARY_PATH for their Python driver to find them. To make matters worse, Apple’s CLI tools strip that variable away as part of macOS’s System Integrity Protection (SIP) before running a command¹.

Given that DYLD_* environment variables are a known attack vector for Mac malware, that’s a good thing in general. However, sometimes one needs a workaround to get the job done.

You can circumvent this SIP feature by using your own tools, e.g., with the help of Homebrew or compiling them yourself. However, given how pivotal shells are to a UNIX system like macOS, things are more complicated than it seems at first sight.

So, let’s look at how that behavior affects you at different levels of obviousness. For that, we’ll set DYLD_LIBRARY_PATH to the XXX marker and see whether it gets propagated or not:

$ export DYLD_LIBRARY_PATH=XXX

Direct Shell Invocations

In the most straight-forward case, Apple’s /bin/bash, /bin/zsh, and /bin/sh don’t propagate that variable:

$ /bin/bash -c 'echo le env: $DYLD_LIBRARY_PATH'
le env:
$ /bin/zsh -c 'echo le env: $DYLD_LIBRARY_PATH'
le env:
$ /bin/sh -c 'echo le env: $DYLD_LIBRARY_PATH'
le env:

…while Homebrew’s do²:

$ /usr/local/bin/bash -c 'echo le env: $DYLD_LIBRARY_PATH'
le env: XXX
$ /usr/local/bin/zsh -c 'echo le env: $DYLD_LIBRARY_PATH'
le env: XXX

So far, so obvious. But as with Docker, the real problem is that it’s easy to accidentally wrap a command into a shell invocation that purges your environment variables unexpectedly.

So, let’s go a level deeper! And the first place that comes to my mind when thinking about indirect command invocations is the realm of build tools like Make or command runners like Just.

Build Tools and Command Runners

If you run an application from your Makefiles or Justfiles, DYLD_LIBRARY_PATH vanishes, regardless whose Make or Just you use.

Therefore, this Makefile:

run:
	echo le env: $$DYLD_LIBRARY_PATH

…gives you even with Homebrew’s Make:

$ /usr/local/bin/gmake
echo le env: $DYLD_LIBRARY_PATH
le env:

That’s because Make uses /bin/sh to run your recipes and that sanitizes your environment variables even if Make doesn’t.

env and Templating Hacks

That means that setting $DYLD_LIBRARY_PATH inside of the Makefile does not work either – /bin/sh will remove it.

Note the double dollar sign $$ in the recipe: It escapes $ and passes the string $DYLD_LIBRARY_PATH to the shell to run. That’s different from writing ${DYLD_LIBRARY_PATH}, where the value of the environment variable is passed.

Thus, while convoluted, you can make it work by adding an explicit env call to the recipe:

# Can't use echo directly because it's a shell builtin.
convoluted:
	env DYLD_LIBRARY_PATH=${DYLD_LIBRARY_PATH} \
		/usr/local/bin/bash -c 'echo le env: $$DYLD_LIBRARY_PATH'

This works (with non-Apple tools!) because the value of the variable is part of the recipe using Make’s string templating and not shell-based variable substitution.

The effective call becomes:

$ /bin/sh -c "env DYLD_LIBRARY_PATH=XXX /usr/local/bin/bash -c 'echo le env: \$DYLD_LIBRARY_PATH'"
le env: XXX

This is ugly, but admittedly, this is also how I made do until I went down this rabbit hole.

Configuring the Shell

A much better solution is to change the shell³ that Make uses to run the recipes to Homebrew’s Bash:

SHELL := /usr/local/bin/bash

run:
	echo le env: $$DYLD_LIBRARY_PATH

As expected:

$ /usr/local/bin/gmake
echo le env: $DYLD_LIBRARY_PATH
le env: XXX

Since macOS strips the variable before Make sees it, this trick doesn’t help with Apple’s Make:

$ /usr/bin/make
echo le env: $DYLD_LIBRARY_PATH
le env:

To achieve the same in Justfiles, use the following:

set shell := ["/usr/local/bin/bash", "-c"]

I’ve switched to set shell := ["fish", "-c"] myself because Apple doesn’t ship the Fish shell, everyone in our company uses Fish, and these Justfiles are only used in development.

So far, we’re still in somewhat obvious territories: commands that run other commands and wrap them in configurable shells. Once you realize what’s happening, it’s straightforward to reason about it. However, it gets more complicated with hidden calls to shells.

Bashes All The Way Down

asdf is a wonderful tool to install various versions of various tools simultaneously. It’s a unified interface for language-specific tools like python-build or node-build. For example, I use it to install Python, Go, Ruby, Node, and direnv.

To decide what version of a tool you get when you run – say – python in a terminal, asdf installs so-called shims for all tools it’s managing and adds them to your PATH. And this is how the python shim looks like:

#!/usr/bin/env bash
# ...
exec /Users/hynek/.asdf/bin/asdf exec "python" "$@"

Surprise Bash! And yes: /Users/hynek/.asdf/bin/asdf is a Bash script, too. None of this is to throw shade on asdf – POSIX shells are supposed to be the glue of a UNIX system.

But it drove me nuts while trying to find out why every Python I had on my system behaved differently.

Apple’s doesn’t work as expected:

$ /usr/bin/python3 -c "import os; print(os.environ.get('DYLD_LIBRARY_PATH'))"
None

Homebrew’s does:

$ /usr/local/bin/python3 -c "import os; print(os.environ.get('DYLD_LIBRARY_PATH'))"
XXX

asdf’s does too, if called directly from the installation:

$ ~/.asdf/installs/python/3.11.1/bin/python -c "import os; print(os.environ.get('DYLD_LIBRARY_PATH'))"
XXX

But it does not, if called via shim:

$ ~/.asdf/shims/python3.11 -c "import os; print(os.environ.get('DYLD_LIBRARY_PATH'))"
None

The shim is what you get when you run a vanilla python after configuring asdf using asdf global python or ‌.tool-versions. That includes the direct execution of scripts with shebang lines without an explicit path – including the popular #!/usr/bin/env python3.

In practice, if you use asdf via direnv as I do, this is not an issue because the virtual environments underneath the .direnv directory don’t point to the shims, but to the installations:

$ readlink .direnv/python-3.11.1/bin/python3.11
/Users/hynek/.asdf/installs/python/3.11.1/Library/Frameworks/Python.framework/Versions/3.11/bin/python3.11

pipx does the right thing too, which means that running my tests with Tox always worked for me:

$ readlink /Users/hynek/.local/pipx/venvs/tox/bin/python3.11
/Users/hynek/.asdf/installs/python/3.11.1/Library/Frameworks/Python.framework/Versions/3.11/bin/python3.11

At this point, we’ve got implicit-but-predictable shell invocations, and we’ve got hidden shell invocations. Let’s up the ante once more and look at hidden-and-unpredictable shell invocations!

Conditional Shell Wrapping

What’s worse than finding a random /bin/sh in your call graph? Sometimes finding a random /bin/sh in your call graph, of course.

The underlying problem is that there are limits to shebang lines (like length or special characters), and workarounds must be employed to adhere to them. One such case that’s easy to test yourself involves Python apps and spaces in path names.

First, let’s create a virtual environment called no_spaces and look at an application in it. We’ll use pip, because it’s always there and doesn’t need to be installed:

$ python -m venv no_spaces
$ cat no_spaces/bin/pip
#!/Users/hynek/tmp/no_spaces/bin/python3.11
# -*- coding: utf-8 -*-
import re
import sys
from pip._internal.cli.main import main
if __name__ == '__main__':
    sys.argv[0] = re.sub(r'(-script\.pyw|\.exe)?$', '', sys.argv[0])
    sys.exit(main())

The shebang line points to the Python interpreter within the virtual environment.

Now let’s try the same thing, but call the virtual environment has space which introduces a space in the interpreter path:

$ python -m venv "has space"
$ cat "has space/bin/pip"
#!/bin/sh
'''exec' "/Users/hynek/tmp/has space/bin/python3.11" "$0" "$@"
' '''
# -*- coding: utf-8 -*-
import re
import sys
from pip._internal.cli.main import main
if __name__ == '__main__':
    sys.argv[0] = re.sub(r'(-script\.pyw|\.exe)?$', '', sys.argv[0])
    sys.exit(main())

We get the identical Python script, but it’s wrapped in a /bin/sh script to avoid having a space character in the shebang line⁴! Therefore, your DYLD_ variables get stripped (or not) depending on the location of the Python interpreter and, thus, the virtual environment.

This problem is not unique to Python – every interpreted language has to deal with these UNIX idiosyncrasies.

The workaround in Python is to invoke your scripts using the python -m form. Therefore, instead of pip install pytest you run python -m pip install pytest. Instead of pytest you run python -m pytest. Et cetera.

There’s already plenty reasons to invoke Python apps using python -m; this is but a cherry on the cake.

Finally, since we’re talking about Python already, let’s see how you can shoot yourself into your DYLD_LIBRARY_PATH with the subprocess module.

Python

Even with ostensibly pure Python, it’s easy to get this wrong by passing shell=True to subprocess.run().

Let’s take this short script that dumps the DYLD_LIBRARY_PATH variable along with its first argument. If the first argument is launcher, it additionally runs itself (__file__) using the current interpreter (sys.executable – in this case: Homebrew’s) once with shell=False and once with shell=True:

#!/usr/local/bin/python3

import sys, os, subprocess

print(sys.argv[1], os.environ.get("DYLD_LIBRARY_PATH"))

if sys.argv[1] == "launcher":
    # Takes a tuple of strings, shell=False is default.
    subprocess.run((sys.executable, __file__, "no"))

    # Shell mode takes ONE string, passes it to a shell.
    subprocess.run(f"{sys.executable} {__file__} yes", shell=True)

Running it gives you the following:

$ chmod +x script.py
$ ./script.py launcher
launcher XXX
no XXX
yes None

The variable exists in the launcher and is propagated into the subprocess if you use shell=False, but vanishes if you wrap the call into a shell by passing shell=True.

As with Docker’s shell form for ENTRYPOINTs, this form of calling run() is more convenient at first sight, but comes with a bunch of surprising side effects. In many ways, it’s an attractive nuisance with redeeming qualities that make it worth having, but it should not be your default choice. os.system() has the same problems, but please never use it.