Optimal Syntax for Python Decorators

Introduction

An optimal decorator syntax

Like all programming languages, Python has many conventions. One of the
strongest is that a series of statements will be evaluated linearly; that is,
from top to bottom and left to right. Decorators are a distinct break from
this linear imperative style and deserve a distinct syntax.

We argue that the current syntax, while quite distinct, is not the
optimal syntax. We present an alternative syntax, and analyze its two most
important characteristics: the use of an indented suite, and the preference
of a keyword over punctuation for the required new token. We finish by
recommending a keyword.

There are many details regarding the implementation of decorators which
will not be addressed in this proposal, because they are orthogonal or
extraneous to the core arguments. Details and issues which do not directly
relate to the two proposed syntax changes are outside the scope of this
document. A short list of extraneous issues includes:

Order of execution of decorators: foo(bar(func)) versus bar(foo(func))

Location of decorators before def (same for both syntaxes)

Whether to allow multiple decorators per line

Backwards portability

The internals of decorator callables

Beauty, Fear, Magic or Intuition

Any semantic or syntactic issue which is not addressed in this proposal
retains the same syntax or semantic as in the pie-syntax implementation
(current CVS, Aug 25, 2004).

Apologies to Guido: this is a dual-purpose document. It has two audiences
and two goals, because time is short. The first audience is the Python
community. However, because the second audience is you yourself, I apologize
for the number of times I quote you and speak about you in the third person.
;)

A Persistent Alternative Syntax

The current syntax, "pie-before-def" 2,
received strong opposition from portions of the Python community beginning
with its inclusion in Python 2.4a2. An example of this syntax:

Dozens of syntax variants were proposed before and since that time; they
are summarized on the Python decorator wiki 3.
Recently, however, the syntax known as "J2" (or "def suite") has arisen as
an option which the general Python community can support. An example of
this syntax:

Unfortunately, no implementation was available for the J2 syntax at the
time of the 'bake-off' request.

No part of J2 has been rejected outright by our BDFL.

Unlike most of the other syntax variants, J2 has consistently dodged
the pronouncement bullet. There was a significant period of time in which
Guido expressed his dislike for a new keyword; however, he has since stated
more than once that a keyword might be allowed given a __future__ import:

"I really don't care if everybody thinks it [pie-syntax] is ugly.
I do care to find out about usability issues... I also want to find
out about superior syntax proposals (from __future__ import decorators
might be acceptable)." 6

No substantive arguments against J2 have been found, as we show below.

Arguments

I. A suite is better than multiple @lines.

1. Rationale

One of the most powerful expectations present in imperative (state-based)
languages is that the vast majority of code will be evaluated at runtime
in the same order in which it appears on the page. Python is no exception.
However, this is no way to write complete programs; conditional branching,
loops, and subroutines have long provided alternative flows of control.
That is, they cause run-time order to be different than compile-time order.

In Python, these "control flow" constructs (if, for, while, try, funcdef,
and classdef) 7 all belong to the lexical
category of "compound statements".
They may utilize dependent "helper statements" (break, continue, raise,
return and yield). These six compound constructs all use the colon as a
delimiter to introduce a indented suite.

In the same way, decorators provide an alternative flow of control. They
cause run-time order to differ from compile-time order. It is therefore
natural and beneficial to adopt a decorator syntax which is similar to that
of the existing flow-control constructs. We therefore recommend that
decorators adopt a grammar which includes an indented suite.

There are four existing constructs which provide multiple suites: if/elif/else,
for/else, while/else, and try/except/finally. All of these derive strong
denotational weight from their indentation. Specifically, although the suites
are indented, the multiple declarations themselves are not. This visually
links the declarations, strengthening their relationship and dependency.
None of the dependent suites (elif, else, except, or finally) "stand alone"
because they would have no meaning if they did so. Outside of the functions
they affect, decorators also lose both denotational and operational semantics.
Indeed, if the multi-suite pattern is not followed, decorator referents are
rendered ambiguous (in the sense that they do not follow any precedent). We
therefore recommend that decorators also follow the multiple-suite pattern.

2. Objections

Objections to the use of an indented-suite syntax have been raised.
One of the first was that "compound statements imply a nested namespace."
Decorator-suites certainly would introduce a new kind of construct,
the first construct wherein one dependent suite does not introduce a
namespace (decorator), while the second suite does (def). However, two
factors mitigate this issue. First, compound
statements are currently ambivalent regarding namespaces; only two of the
six introduce new namespaces (funcdef and classdef). Second, namespaces are
wholly dependent upon control flow suites. The addition of a mixed-namespace
construct can be seen as a refinement of the policy of suites in general,
not a departure. As Guido put it when discussing thunks,

"Many users have only a vague notion of the difference between a
syntactic construct and a function (witness repeated suggestions here
and elsewhere that confuse these matters), yet nobody has a problem
with understanding the difference in scope rules between 'def' and
'if' statements, for example." 8

The second major objection is that a suite for decorators violates the
expectation "that a suite is a sequence of statements"
9. This is true;
however, normal Python code shares that expectation. Therefore, the current
pie-syntax shares this weakness. Further, it is almost exclusively within
suites that one currently finds bare expressions which are not statements.
For example,
docstrings are most often found within function and class suites. Lambdas
are another common example, albeit degenerate, of bare expressions in a
suite-like structure (although lambdas do not officially possess 'suites'
in the Grammar, they possess the delimiting colon, and are therefore
denotational siblings of one-liner suites).
Given these precedents, a suite is a more appropriate and consistent place (than
normal linear code) in which to introduce a series of decorator expressions.
Finally, although the decorator expressions appear in code to be
declarative expressions, they behave at runtime like imperative
statements.

Last, some have said the use of a suite is "too heavyweight" when used
for a single decorator, the most common case. The use of a suite does use
more characters (and more typing). We answer by pointing to exceptions as a
precedent; it has been generally-accepted programming practice in the Python
community for some time now to limit the first suite of a try/except block
to a single statement ("narrow the exception"). Regardless, we find the
extra whitespace a feature to embrace, rather than avoid (see below).

3. Additional benefits

Some additional benefits arise from the use of suites. First, as mentioned above, a suite
strengthens the visual association between the decorators and the referent
function. A paired, compound statement of <token>: and
def: strengthens the fact that both are declarations; further,
they each produce a new binding within the current scope.
Note, in particular, that the result of a series of decorators is a single
new binding. Pie-before-def syntax does not make this as clear as does a
suite. Although the decorator suite does not introduce a new namespace,
it may hint that some namespace trickery is going on--that any intermediate
bindings are discarded.

Second, it is easier for humans to visually distinguish the location of
the function signature with a decorator suite (when multiple decorators are
present). It could be argued that another de facto expectation exists
for Python: that flow-control suite headers are preceded by either a blank
line or an associated, indented suite.

One drawback to the suite version is that the actual name of the
function might become obscured, since most editors indent 4 spaces,
causing the decorators, the function name, and the function code
block to line up in the same starting column.

Third, a decorator suite strengthens the visual association of the
decorators to each other, delimiting them both vertically (via one
identifier for the whole set as opposed to one each) and horizontally
(via shared indentation).

Fourth, source-code folding is easier with a suite.

Finally, the use of a suite for decorators provides greater flexibility
for the future. Once decorators begin to be used in earnest, people will
undoubtedly find new ways to use them, and will begin to want more power
from decorator syntax 11. Some of these uses
have already been put forward:

Docstrings

Function attributes

Application to classes or future thunks as well as def's

Specification of alternate targets (a la C#), such as the return
value of the decorated function

This proposal does not possess the scope or intent to detail every
possible future for decorator semantics. However, the use of a suite for
decorators gives a clear benefit to designing the syntax of those possible
futures. Because suite-syntax uses existing syntax structures, other
existing syntax can be more easily adapted to the decorator model in future
releases of Python. The suite provides all of the semantic power of
statements, without having to tag each decorator with its own 'decorator
token'. Perhaps most powerfully of all, decorator suites could someday
allow a mixture of decorators and normal Python statements within the same
suite, without consequently making the referent of each decorator ambiguous.

II. A keyword is better than punctuation for a new token.

The choice whether to use a suite for decorator statements is an important
one. Equally as important is the choice between punctuation or keyword for the
required new syntax.

1. No new token category

First, browsing through the Language Reference, one is immediately
struck by the simplicity of the lexicon. Besides whitespace, tokens are
either identifiers, keywords, literals, operators, or delimiters.
The @-prefix for decorators is unfortunately none of these. Prefix
punctuation appears in only two other places in standard Python: *args
and **kwargs, and the "reserved classes of identifiers", those with
leading underscores 12. Pie-syntax is
certainly not a reserved class of
identifiers, and mysteriously, *args and **kwargs are not mentioned in the
section on lexical analysis at all. In short, prefix punctuation for
decorators requires the development of an entirely new category of token.
The use of a keyword, by contrast, uses existing syntax to provide a new
token without a new category.

2. Matches existing use of tokens

Second, keywords have traditionally been used to express control flow
in Python. All of the existing flow-control statements, both compound
and simple, use keywords. As the authors of PEP 255 (Simple Generators) put
the matter when discussing 'yield':

Q. Why a new keyword for "yield"? Why not a builtin function instead?
A. Control flow is much better expressed via keyword in Python, and
yield is a control construct. 13

Punctuation, by contrast, is used almost exclusively for operators and
delimiters. The use of punctuation in the context of decorators would,
we believe, break these long-standing conventions and harm readability
14.
In addition, it would restrict future uses of a limited set of punctuation:

"On the plus side, Java's @name(kwargs) syntax allows us to
put decorators in front methods and classes without ambiguous syntax;
on the minus side, using up a potential operator character for one
specific purpose should not be done lightly."
15

In other words, we are in far more danger of running out of punctuation
(and therefore, new tokens for operators and delimiters) than we are in
danger of running out of keywords (and therefore, new tokens for control
flow constructs).

3. Enforces necessary change control

"A prefix syntax should be designed with future
usability in mind exclusively, unconstrained by "implementability without
language changes". That's a much more stifling requirement than the
requirement that existing code shouldn't be broken (which is stifling
enough in itself but IMO unavoidable)."
--Guido van Rossum

Third, new punctuation does not enforce code-level structure to manage
the change process. At least two developers (of Leo and IPython
16) have
already come forward, stating that their tools will have to be modified to
accomodate the proposed @ symbol for decorators. The unfortunate reality
of large-scale open-source development is that we cannot know in advance
how many more such cases exist; without code-level change control, we are
helpless to mitigate the risk. A new keyword, however, would be required to
take advantage of a __future__ declaration 17,
which provides an established,
well-known, successful mechanism for change control.

In particular, __future__ provides a more measured release schedule.
During the interim (when "from __future__ import decorators" is required
to use decorators), developers gain the ability to apply the new syntax
on an "as-needed" basis; that is, they can use the new syntax in one
application while delaying the syntax in another. Where __future__ is not
invoked, the keyword will continue to function as a normal identifier.
For example, one developer may have immediate need for decorators,
while another may need a substantial amount of time to migrate their code.
Admittedly, the latter would only face issues under the pie-syntax if using
Leo, IPython, or some other nonstandard use of @. Again, however, we are not
aware of these other uses, whether of @ or a keyword token. The use of
__future__ mitigates this risk.

Finally, the use of __future__ provides code-level documentation of the
change process. From PEP 236:

"To document when incompatible changes were introduced,
and when they will be--or were--made mandatory. This is a form of
executable documentation, and can be inspected programatically via
importing __future__ and examining its contents."
17

4. Python decorators are not Java annotations nor .Net attributes

Python decorators have been through several transformations. Most
recently, decorators adopted syntax from C#, and then, Java. In particular,
this similarity of syntax has been put forward as a benefit in favor of
these syntaxes; specifically, that users of these other languages would
have an easier time deciphering Python decorators, based upon their
familiarity with the syntax in the other language(s).

The current @-decorator syntax is inspired by Java's "annotations".
In fact, the authors of the Java annotation feature state that the @-symbol
is a mnemonic for annotation type. However, Python decorators
differ significantly from Java's use of that syntax. Most importantly,
"annotations" are limited to attributes and other non-transformative
operations in Java. From the JDK 1.5 specification:

"This facility allows developers to define custom annotation
types and to annotate fields, methods, classes, and other program elements
with annotations corresponding to these types. These annotations do not
directly affect the semantics of a program." 18
(emphasis added)

Java annotations provide semantic transparency; the referent is
not modified. One may use "interceptor" functions like @trace, but the
annotated function is not wrapped. In Java, function transformations continue
to be provided by keywords such as static.

In contrast, all Python decorators accept one function and return one
function. Many of them will return a new function object. Many will change
function signatures, affecting both the decorated function and its callers.
The most common decorators, classmethod and staticmethod, are among these.
In short, Python decorators propose the opposite implementation of
Java's annotations.

Similarly, the previous []-before-def syntax was inspired by Microsoft's
C# language. But C# also limits the expressivity of what it calls "attributes".
They are similar to Python decorators in the sense that both involve what
appear to be expressions as opposed to statements; however, .Net attributes
really are non-statements; they do not modify their referents so much as
attach themselves to them. Microsoft often calls attributes "declarative"
as opposed to "imperative" 19. Further,
they are required to be commutative 20,
a requirement which would cripple many uses of Python decorators
21.

Both syntaxes (those inspired by C# and Java) suffer doubly: although they
promise semantic parallelism, they not only fail to deliver it, but deliver
the opposite semantic. A keyword syntax is free to define its own semantics
from scratch, and will lessen the confusion of developers who are new to
Python, yet experienced in these other languages.

III. Choosing a keyword

If a keyword is to be chosen over @ or other punctuation, the question
remains, "which word should it be?" So many words have been proposed that
it would be difficult to examine each one in turn. Therefore, we begin by
establishing guidelines for the selection of a keyword. Such a keyword:

Should not be used widely or critically as an identifier in
existing Python code.

Should be easy to remember when writing new code.

Should be easy to remember when reading existing code.

Should be easy to search for, in both docs and Google.

Should not be a word with a planned future of different semantics.
This rules out "as", and probably "with". 22

Should not be "def", which Guido rejected as "too weird. The
stutter doesn't have any semantic connotation to it and is bound to
confuse source-scanning tools." 23

Should either pair strongly with "def", or at least not stand alone.

Should not address only one aspect of decorators.

Should not be a form of the word "decorate". The term "decorate"
conflicts with two separate concepts: both the GoF Decorator pattern
(which is a runtime wrapper, not a compile-time one), and with our
own beloved "decorate-sort-undecorate" pattern (aka Schwartzian or
Guttman-Rosler Transform).

Candidates for keywords have fallen into several camps. Many of the more
common candidates suffer by addressing only one aspect of decorators. Prime
examples are keywords such as "transform" and "wrap", which refer to the
nature of those decorators which return new function objects; such keywords
do not fare well when applied to decorators which merely annotate. Conversely,
the words "declare", "decorate", and "decor" address the annotative aspects of
decorators, ignoring the transformative aspects. In addition, such words
would tend to clash with the names given to decorators themselves; for example,
the test suite for decorators names a decorator function "decorate".
It isn't difficult to imagine users wanting decorators named "transform"
and "wrap", as well, even if such generic names are discouraged.

As discussed eariler, decorators have no meaning apart from their referent
functions. Indeed, this is a strong part of the case for a suite syntax,
which more strongly denotes the dependency. We recommend that a keyword be
chosen which directly refers to this dependency. Some examples:

Of these, the words "using", "as", "per", and "predef" have each had
multiple advocates within the community.

Looking again at existing control flow constructs, we notice that keyword
choice follows purpose. "Try" is a verb and its object is the suite. "Def(ine)"
is a verb, and its object is the suite 24.
"If", "while", and "for" all refer to their declarations or tests, not
their suites, and function adverbially. For example, in the statement
"while (x == 3)", the object of the keyword "while" is "x == 3",
not the suite of statements which follow. More importantly, the dependent
clauses (else, except, and finally) all function adverbially. Since decorator
suites are also dependent and also function adverbially, the keyword for
decorator suites should either be an adverb, or a preposition or conjunction
which is often used adverbially.

Of the candidates, we recommend the keyword "using", because:

It functions as an adverb.

It emphasizes the dependent nature of decorators because it
cannot "stand alone" semantically.

It emphasizes the generic nature of decorators.

It "reads correctly", forming rough English in every use case. Example:

It does not clash with existing Python code.
In the Library, webbrowser.py has a function with the signature,
"def get(using=None)". No project has been found which calls this
function using keyword args.
Other major projects checked which do not clash: Chandler,
wxPython, libxml2, SWIG, PyXML, ZODB4, distutils (as a standalone),
egenix-mx, Redfoot RDF tools, PyLucene, dbxml, libxslt, jabber-py,
PyEgads, PyChecker, soappy, m2crypto, epydoc, 4suite

Conclusion

We began by stating that a distinct new language feature deserves a
distinct syntax, and pie-syntax is certainly distinct! We have shown that
alternatives exist, particularly, that the use of a suite, and the keyword
"using" for the new token, supply a new, distinct construct which better
fits the precedents of existing Python. In the absence of further technical
arguments, the "J2" syntax results in decorators which are more conventional
and readable, yet also more extensible and manageable. In contrast, the
current "A1" syntax is more radical and also more limited. We leave the
decision between these outcomes to our trusty BDFL.

Signatories

If you wish to add your name to this proposal, either for, against, or
abstaining, please send mail to either comp.lang.python or the author
(fumanchu@amor.org). If you reject
the proposal, please include a summary of your objections so that your
opinion can be included toward Guido's more complete understanding.

In favor, stating that the proposal as written provides a more
acceptable syntax than the current syntax, and that part or all
of the Arguments should be implemented,

"I've got a little bit of philosophy on the use of keywords vs.
punctuation. While punctuation is concise, and often traditional for
things like arithmetic operations, giving punctuation too much power
can lead to loss of readability. (So yes, I regret significant
trailing commas in print and tuples -- just a little bit.)"