Please Fix Your Decorators

If your Python decorator unintentionally changes the signatures of my callables or doesn’t work with class methods, it’s broken and should be fixed. Sadly most decorators are broken because the web is full of bad advice.

(Feel free to jump directly to the conclusion if you just want to know what to do.)

Quick refresher

A function decorator¹ is a way to wrap callables (i.e. mainly functions and methods) to change their behavior. Common reasons are executing code before and/or after the wrapped callable, modifying arguments or return values, caching, and so forth.

They are callables themselves that are applied to the wrapped function:

def func():
    ...

func = decorator(func)

Since this is a very popular pattern, Python has syntactic sugar for it and so

@decorator
def func():
   ...

is identical to the first listing.

Now let’s take the example from the Python documentation:

from functools import wraps

def my_decorator(f):
    @wraps(f)
    def wrapper(*args, **kwds):
        print('Calling decorated function')
        return f(*args, **kwds)
    return wrapper

…and see what’s wrong with it. For that we define a useless function that multiplies two numbers but allows for an optional third factor that is 1 by default:

def mult(a, b, c=1):
    return a * b * c

mult_decorated = my_decorator(mult)

The problems

Mangled signatures

Let’s inspect mult using the official Python 2/3-compatible function (it even got un-deprecated in 3.6):

>>> import inspect
>>> inspect.getfullargspec(mult)
FullArgSpec(args=['a', 'b', 'c'], varargs=None, varkw=None, defaults=(1,), kwonlyargs=[], kwonlydefaults=None, annotations={})

You can see the names of all arguments and the default value for argument c.

How does it look like for the decorated version?

>>> inspect.getfullargspec(mult_decorated)
FullArgSpec(args=[], varargs='args', varkw='kwds', defaults=None, kwonlyargs=[], kwonlydefaults=None, annotations={})

Wow, the information is all but gone: the signature of the wrapper closure became the signature of mult! But why!? We used functools.wraps!

Turns out, functools.wraps only preserves

1. the name
2. and the docstring

of the wrapper callable:

>>> mult.__name__ == mult_decorated.__name__
True
>>> mult.__doc__ == mult_decorated.__doc__
True

which is nice but not enough (it also sets the __wrapped__ attribute; more on that later). If you feel stupid now: don’t. Even the smartest of us stepped into this trap.

This behavior is an annoyance at best since broken introspection will break things like debugger introspection or argument auto completion in your fancy Python shell:

But it also has serious consequences at worst, if you use a framework or library that uses the signatures of your callables for useful things.

The place it bit me was Pyramid behaving differently depending on the argument count: your views get a context object additionally to a request object if it “fits the argument list”. Of course, it fits into *args, **kw. But it may not fit into your view if it has only one argument. So you get a cryptic TypeError about some context thingy you’ve never heard of and are left puzzled.

And if everything breaks in puzzling ways just because I’ve applied an innocent-looking decorator, I can’t wait for the return of Cthulhu.

Class methods

Next up, we try our decorator on a class method². I use class methods quite a bit for (often asynchronous) factories so I really want my decorators to work with them:

class C:
    @my_decorator
    @classmethod
    def cm(cls):
        return 42

Let’s call C.cm()!

>>> C.cm()
Calling decorated function
Traceback (most recent call last):
  File "<input>", line 1, in <module>
    C.cm()
  File "<input>", line 5, in wrapper
    return f(*args, **kwds)
TypeError: 'classmethod' object is not callable

Ouch³!

I’m gonna stop at this point because I feel it’s been enough to get my point across. The sad fact is though, that there’s many more corner cases that you and I can’t even think of.

So if you’re writing software that is used by other people, you should be aware of that and consider one of the following remedies.

The solutions

Python 3.5 and inspect.signature()

This is where the aforementioned __wrapped__ attribute comes into play: as of Python 3.5, the inspect.signature() function together with functools.wraps will do the right thing by following it and returning the “real” signature:

>>> inspect.signature(mult_decorated)
<Signature (a, b, c=1)>

Upsides:

• You don’t have to do anything. ✨

Downsides:

• Doesn’t fix class methods.
• inspect.signature() is Python 3 only and only does the right thing on Python 3.5. You can simulate the correct behavior on Python 3.4 by calling inspect.unwrap() on your wrapped function.
• It’s kind of cheating: the signatures are still broken. inspect.signature() is just smart enough to lie to you.
• All in all, your package is broken on anything except Python 3.5+ which makes it a poor choice for public libraries.

boltons

I’ve come to use Mahmoud Hashemi’s boltons package in almost every project nowadays.

And its funcutils module contains an implementation of wraps that preserves the signature of your callable.

For better or for worse, its version of wraps already explodes when even defining a class method:

Traceback (most recent call last):
  File "<input>", line 1, in <module>
    class C:
  File "<input>", line 3, in C
    @classmethod
  File "<input>", line 2, in boltons_decorator
    @funcutils.wraps(f)
  File "/Users/hynek/.virtualenvs/tempenv-414e4537272f/lib/python3.5/site-packages/boltons/funcutils.py", line 282, in wraps
    fb = FunctionBuilder.from_func(func)
  File "/Users/hynek/.virtualenvs/tempenv-414e4537272f/lib/python3.5/site-packages/boltons/funcutils.py", line 466, in from_func
    kwargs = {'name': func.__name__,
AttributeError: 'classmethod' object has no attribute '__name__'

But it will fix the signatures without rewriting any code. Just use the wraps function from boltons.funcutils instead of the one from functools.

While boltons is a collection of utilities, you can use each sub-module in isolation and simply drop the module into your own project.

decorator.py

The next easiest way is the single-file decorator.py package. It has no dependencies itself and authors are encouraged to vendor it by just dropping it into their own projects if they want to avoid dependencies too.

One of its downsides is that you have to rewrite your decorators. I like its closure-less style better though:

from decorator import decorator

@decorator
def decorator_decorator(f, *args, **kw):
    print('Calling decorated function')
    return f(*args, **kw)

As with boltons, decorator.py doesn’t work with class methods and fails in a very similar way.

wrapt

wrapt does all of the above and much more. Broadly speaking, it takes care of all the corner cases you haven’t even heard of. It too will require you to rewrite your decorators:

import wrapt

@wrapt.decorator
def wrapt_decorator(wrapped, instance, args, kw):
    print('Calling decorated function')
    return wrapped(*args, **kw)

Again, I like its style better. Now let’s take it for a ride:

class C:
    @wrapt_decorator
    @classmethod
    def cm(cls):
        return 42

And lo and behold:

>>> inspect.getargspec(wrapt_decorator(mult))
FullArgSpec(args=['a', 'b', 'c'], varargs=None, varkw=None, defaults=(1,), kwonlyargs=[], kwonlydefaults=None, annotations={})
>>> inspect.signature(wrapt_decorator(mult))
<Signature (a, b, c=1)>
>>> C.cm()
Calling decorated function
42

Upsides:

• The most correct of them all – period.
• Does a lot more related tasks.

Downsides:

• Slowest of them all despite of an optional C extension. Correctness has its price at runtime.
• Decorated functions are not pickleable.

Conclusions

1. I use wrapt in all my own code because I value correctness and lack of surprises above all.
2. If you need pickleable functions, or you want minimal (or no) dependencies, and only care about having working signatures, the next best solution is either boltons or decorator.py – depending on which style you prefer. I prefer boltons.
3. If you decide to not care about anything I’ve presented, you at least know what to look for if stuff breaks now.

Credits

All of this (and more!) has been previously explained in a much more elaborate and technical fashion by Graham Dumpleton before (2014!). I can’t recommend enough the whole blog series about this topic if you’re interested in this kind of stuff.