TIL: rq & macOS

Due to fork() semantics, using rq on macOS can lead to confusing crashes.

For instance, httpx uses internally urllib.request.getproxies() from the standard library which makes the process crash with

objc[51435]: +[__NSCFConstantString initialize] may have been in progress in another thread when fork() was called.
objc[51435]: +[__NSCFConstantString initialize] may have been in progress in another thread when fork() was called. We cannot safely call it or ignore it in the fork() child process. Crashing instead. Set a breakpoint on objc_initializeAfterForkError to debug.
Moving job to FailedJobRegistry (work-horse terminated unexpectedly; waitpid returned 6)

The best solution in this concrete instance is telling Python to not look up proxies and run rq with the following environment variable set:

export NO_PROXY=*

If that’s not enough, you can disable the safety check altogether:

export OBJC_DISABLE_INITIALIZE_FORK_SAFETY=YES

Be aware that this is akin to taking the batteries out of your smoke detector to stop the annoying beeping. fork() should only be used if you immediately exec() afterward – if at all.

This is also the reason why multiprocessing is not using fork() by default on macOS since Python 3.8.

See also: How macOS Broke Python