A Stackless PEP

08/23/2000

Gordon McMillan will be writing a Python Enhancement Proposal (PEP) for
making the core Python interpreter stackless. A stackless interpreter
for Python is an arcane concept. Maybe that is why it fascinates
me. It appeals to the geek in me. Stackless is a project to untangle the
Python interpreter from the C stack. C as in the C programming language;
stack, as in a list onto which you push or pop things. Let me try to untangle
that for you some.

The C stack tracks the execution of functions in a C program. When each
new function is called, a pointer to the function and its state information is pushed onto the
stack. When the new function terminates, the old function is popped off and the interpreter resumes processing it.

This is a pretty straightforward way of handling things for your average
C program. Python, however, tracks this kind of execution itself in
something it calls a Frame. Well, it almost does it itself -- it relies heavily on the C stack which maintains pointers to each Frame as it executes. This seemed like a good thing at the time. There are some
problems with the current arrangement, though.

The C stack size is fixed; that limits the depth of functions you can
call in a Python program. That can be a big issue when you are using recursion,
a function calling itself repeatedly. Getting rid of the size limitation
is one benefit of Stackless, but the real benefit comes when you want to
manipulate the stack itself. When the Frame is decoupled from the C stack, you can do some tricky things, like hopping between functions on a stack and
passing information between those functions, or maintaining multiple stacks and
swapping them about. Stackless can be used for coroutines, a fast
alternative to threaded programming; it makes state machines a cinch; and it makes it possible to use continuations, something Christian Tismer describes as "the most
powerful control structure known in information science" and MacMillan refers to as "brain-exploding stuff." We will take a closer look at what you can do with Stackless in an upcoming series of brain-exploding articles.

Stackless Python is
a bit of a misnomer. It still uses the C stack, it just separates
Python's Frames from the C stack, making all these fancy tricks possible. It also isn't recent news. You can make Python stackless now. Christian
Tismer maintains the stackless project and does what he can to keep his
patches to make Python stackless current with the development of Python
itself. Keeping up with Python development takes time, though. It's
time Tismer could be putting into further enhancing Stackless Python. Many
people have said Python's core should be stackless. If it is part of the
core, then Tismer no longer has to play catch up. Gordon McMillan's PEP
might actually make it happen.

Stackless may not be new, but PEPs are. They are a Python specific Request
For Comments (RFC). They formalize the process of suggesting changes to
Python. If you want to suggest a change, you write up your arguments and present
them as a PEP. Those arguments are discussed, and eventually the PEP is
either accepted or rejected. During the last round of calls for Python going
stackless, the reply from the development team was very clear: "Write a
PEP."

So that's what Gordon McMillan is going to do for Stackless.
You can find an abstract for his PEP and many other PEPs on Python at
Sourceforge.

If you are an avid user of Python, you should keep an eye on the PEPs.
By commenting on them, you will help shape the future development of the
language you use. The two PEPs that have inspired the most discussion recently
have been PEP 211, which calls for new operators for performing linear
algebra, and PEP 214, a suggestion for extending the standard print statement.
Stackless is PEP 219 and is currently no more than an abstract. McMillan says it may take a few months to complete. To make including Stackless more compelling, the Stackless group wants to complete a simple API for creating coroutines. When the coroutines API and PEP draft are ready, it
will either be a no-brainer to add Stackless to the core, or it will be the
next big controversy.