Python Roadmap

How does Stackless Python fit into Python's future? To understand that, we first need to see where the Python programming language is headed.

Lots of news

2000 will have been an eventful year for Python. The biggest news so far has been the westward shift of its developmental center-of-gravity.

Guido van Rossum conceived Python at the end of 1989 while a researcher at the National Research Institute for Mathematics and Computer Science (Centrum voor Wiskunde en Informatica--CWI) in Amsterdam. Since 1995, the not-for-profit Corporation for National Research Initiatives, headquartered in Reston, Virginia, has employed him. At the end of May, though, "open-source applications portal" BeOpen.com hired van Rossum and several colleagues to construct PythonLabs within the Santa Clara-based BeOpen Network. PythonLabs headquarters remains in Reston.

At the engineering level, version 1.5 in its various releases (1.5.2 was the last release of Python 1.5) remains current. There have been two recent follow-ups to 1.5. One is called 1.6, released in it's final form last month. The other is designated 2.0, and was just released this week. Many working programmers will find 1.6 and 2.0 indistinguishable for daily work. Andrew Kuchling, who twice a month summarizes traffic on the python-dev mailing list where deep issues are worked out, puts it this way:

1.6 is the Python CVS tree at the end of May, fixed so that Unicode and SRE [a regular expression alternative] both work and are up-to-date. 2.0 includes all the development since May, which includes new modules, most of the serious language changes such as list comprehensions, and more fixes.

Why have two versions so close in appearance with such different labels? A tangle of reasons, including the relocation from CNRI to BeOpen, led to this slightly confusing resolution. While the inner circle struggles with licensing and numbering complexities, the highlights for external developers are simple:

Python is as free as ever. It can be used, modified, copied, and so on, without restriction. Consult a lawyer as appropriate.

The biggest advantage of Python 1.6 and 2.0 over 1.5 is their complete support of Unicode. Unicode's prominence in XML standards makes this enhancement a crucial one even for programmers with no obvious interest in "internationalization." Also, the core 2.0 distribution now includes several useful new library modules which manage zipfiles, Unicode conversion, memory-mapped files, and more. Other changes are mostly technical: details of type usage, display formats of numbers, and other relatively minor matters.

Python 1.6 and 2.0 are not Stackless. Work at different levels has already begun on both 2.1 and what is lightheartedly called "Python 3000." Python 2.1 will be the evolutionary follow-up to 2.0. Python 3000 will be a substantial rewrite of Python, perhaps the largest one yet, and likely will appear in 2002 or later.

Whether 2.1 includes Stackless or not remains an open question. Should 2.1 incorporate Stackless? As Introduction to Stackless Python, the first article in this series, explained in more detail, not even that most basic question has been
decided yet. Stackless' fans argue that its inclusion is an easy choice. Stackless Python is faster, more capable, and mature, because it's been used for many months in "mission-critical" applications already in production. A change to stacklessness is a "pure win," with no significant costs.

On the other side are two main arguments:

Van Rossum is conservative. That is, he's careful with Python's development. There's no denying that Stackless represents a significant change to one aspect of Python's implementation. Van Rossum simply needs time to digest all its implications, and determine whether it's truly safe for his language.

JPython presents difficulties.

JPython requires a bit of explanation.

JPython complicates the picture

"Normal" Python, the kind that you'd install as 1.5.2 on your desktop or server, is implemented in source coded in C. Python is quite portable, in the sense that it's available on a rich variety of platforms. The same source is used to make executable Python language processors for Unix, Win*, MacOS, OpenVMS, and a variety of other operating systems (OSs). This "mainstream" Python is sometimes called "CPython."

At the same time, there are a handful of alternative implementations of the Python interpreter. Most are used only for research. However, as we've already seen, Stackless Python shares almost all of its implementation with 1.5.2, and is available for almost all of the same OSs. A third widely-used Python implementation is JPython.

JPython is a language processor with very nearly the same functionality as CPython, but implemented in Java. When you command print 1 + 3, both CPython or JPython act the same, in sending 4 to the screen. The difference, though, is that you run JPython within a Java Virtual Machine (JVM). This means, for example, that you can launch a JPython instance in a Java-enabled Web browser and thus execute client-side Python Web scripts. Also, JPython knows about Java; you can create instances of Java-coded classes and invoke Java methods from within JPython.

JPython is important. Large organizations are choosing Java as a standard, and JPython fills a need there and elsewhere.

JPython presents a challenge for Stackless, though. JPython can't be patched with the Stackless change as easily as CPython; in fact, serious Python experts believe that JPython is intrinsically incompatible with Stackless. Deeper technical research should eventually yield a definitive answer to this question. No one questions, though, that at best a Stackless upgrade for JPython will be a delicate and time-consuming task.

Introduction of Stackless therefore will break compatibility between CPython and JPython, at least in the short- and medium-term. Some Stackless fans minimize the significance of this rupture. One of Stackless' strongest evangelists is Just van Rossum, Guido's brother, who wrote in early August, "The Java argument is a very poor one." His counter is that just as JPython goes beyond CPython with its well-modularized ability to load Java definitions, there's no long-term harm in endowing CPython with Stackless's capacity to import continuation.

Microthread specialist Mike Fletcher makes a different point. He observes that, although full continuations are hard for JPython, such higher-level interfaces as coroutines and microthreads can be emulated adequately with JPython threads. If most developers exploit only these APIs to Stackless Python, then there need be no
conflict with corresponding implementations in JPython, as long as as they're run in JVMs with sufficient memory and related resources.

Stackless' use

The control structures Stackless introduces are likely to be more subtle and pervasive than the occasional Java class definition, though. Stackless excites engineers who want to use it. Suppose CPython becomes Stackless, and applications dependent on Stackless continuations begin to proliferate. Is it a net benefit for Python when so many programs won't run with JPython? That's the sort of thorny question now before Guido van Rossum and his core team of advisors.

Averaging together conversations with a dozen leading Python players leads to this "average" of all the prospects: Next year, CPython 2.1 will most likely be rewritten in terms of a mature Stackless coding, that is, one that Tismer has been using for several months already. This gives the implementation advantages of stack protection,
performance, and suitability for such unusual environments as PalmOS. It leaves to future releases a few boundary Stackless issues such as the particular class relation between what Tismer now calls FrameState and FrameData. Also, a specific user-level
coding interface, such as Continuation, CoRoutine, Generator, Microthread, or so on, might not make it into 2.1. This interface is a separate issue from the Stackless surgery that must first be done on the byte-code interpreter. Details of the concurrency programming interfaces would likely be ironed out in time for follow-up versions to 2.1.

Inclusion of some Stackless variant in 2.1 will free Tismer and his colleagues from having to track ongoing Python development with large patches. He'll then invest more of his time on refining Stackless to further improve its performance, persistence, and extensibility. The Stackless cohort will be able to work in parallel at exposing and documenting a programming interface. Once Stackless makes it into the core Python distribution, its use should spread rapidly.

It seems reasonable to expect Stackless will make it into Python 2.1. This is likely to unleash a burst of activity, as higher-level interfaces -- microthreads and coroutines, among others -- are shaken down for 2.2. Stackless is a sufficiently interesting change that it might take many months to grasp all its consequences.
Further clouding the picture is the emergence of Microsoft's .NET technology as an important part of the landscape. .NET appears poised for breakthrough, and the Python .NET interpreter in particular will be crucial in this process. What impact will Stackless and Python .NET have on each other? So far, all the experts will say is that they're working to build them toward each other.