Using Celery with Pyramid

This one falls under: “I knew there has to be an easy way!”

While there’s pyramid_celery, last time I checked it had several shortcomings (back then I couldn’t configure celerybeat) and I felt in general that it shouldn’t be necessary to add another dependency that may or may not lag behind pyramid’s and celery’s latest releases. I also don’t really care for configuring celery inside of an .ini file.

After some tinkering and annoying Ask, I found a very simple solution that worked perfectly for all my needs. Assuming you’re using celery’s 3.0-style API, all you have to do to be able to use SQLAlchemy and access Pyramid’s .ini configuration (whose file name is defined in the env variable YOUR_CONFIG in this case) items is the following in the fictitious your_app/celery.py file:

from celery import Celery
from celery.signals import worker_init


@worker_init.connect
def bootstrap_pyramid(signal, sender):
    import os
    from pyramid.paster import bootstrap
    sender.app.settings = bootstrap(os.environ['YOUR_CONFIG'])['registry'].settings


celery = Celery()
celery.config_from_object('celeryconfig')

It reads the celery configuration from celeryconfig.py and after that, you can access it as simply as:

from your_app.celery import celery


@celery.task
def foo():
    print(celery.settings['sqlalchemy.url'])

Please note that the tricky part here is that you can’t bootstrap Pyramid at import time or you’ll get inevitably into circular import hell.

Hynek Schlawack