TIL: Python Project-Local Virtualenv Management

On UNIX-like operating systems you can have the Python equivalent of node_modules today, for every Python version, without changing your workflows.

There is tool-envy in the Python community towards communities with all-encompassing packaging tools¹ like JavaScript and their npm. One of the commonly desired features is something akin to the node_modules directory instead of managing virtualenvs.

On UNIX-like operating systems², you can have that thanks to a tool that I can recommend independently of Python packaging: direnv.

At its core it’s for setting environment variables when entering a directory. But it has a ton of other features, including baked-in support for venv (and more Python-specific tools).

All it takes is the following line in the .envrc in the project’s directory:

layout python python3.9

Now whenever you enter that directory, direnv will look into the local .direnv directory and either activate or create a venv for the exact Python version. As of writing this it would be .direnv/python-3.9.6.

If something breaks, just delete it. The only thing to keep in mind is that it’s usually a good idea to update pip and setuptools whenever direnv creates a new venv for you (aka whenever entering the directory takes unusually long). To clean up stale venvs I just use

$ find . -type d -name "python-OLD-VERSION" -delete

If you use pyenv to manage your Python installations³, you can use a one-liner to select the right version and activate a venv:

layout pyenv 3.9.6

My thanks go to Chris for telling me about this a few months ago.

Windows users will have to wait for Kushal’s PEP 582 to ship or play with his implementation of it. Or take a look at pdm.

I use the following fish shell function to keep my .direnv clean:

function ,kill-old-venvs
    set CURR (type -p python | sed -E 's/.*(python-3\.[0-9]+\.[0-9]+).*/\1/')
    echo Current Python version: $CURR

    set ENVS (find .direnv -depth 1 -name "python-*" -and -not -name $CURR)

    if [ "$ENVS" = "" ]
        echo "no old venvs"
        return 1
    end

    read -l -P "really delete $ENVS [y/N] " confirm

    if [ $confirm = "y" ]
        rm -rf $ENVS
    end
end