Personal tools

WebOb Now on Python 3

Report about porting to Python 3.

In mid-September, I requested a grant from the Python Software Foundation to
port WebOb to Python 3. The proposal was as follows:

Porting WebOb To Python 3
=========================
Proposal submitted to the Python Software Foundation
September 16, 2011
Abstract
--------
WebOb ( http://docs.webob.org/en/latest/index.html ) is a library used
to make creating Python web applications easier. It contains request
and response abstractions which automate tasks such as setting
response headers, setting cookies, decoding headers and cookies,
generating ETags, and working with query strings. It is based on the
WSGI protocol.
WebOb was created by Ian Bicking, and is now maintained primarily by
Sergey Schetinin.
WebOb is currently used in the following major systems:
- Pylons
(https://docs.pylonsproject.org/projects/pylons_framework/dev/)
- Pyramid (https://docs.pylonsproject.org/docs/pyramid.html)
- Google App Engine (http://code.google.com/appengine/)
- Bobo (http://bobo.digicool.com/)
- Restish (http://ish.io/projects/show/restish)
- TurboGears 2 (http://turbogears.org/2.0/docs/)
It is also used by other less-well-known frameworks and in many WSGI
middleware implementations.
Currently, WebOb is limited to running on the following platforms:
CPython 2.5, CPython 2.6, CPython 2.7, PyPy 1.5 and 1.6, and Jython
2.5.2. This proposal requests funding to port WebOb to Python 3. I
propose to do this work. After the work is completed, a version of
WebOb will be available which is compatible with both Python 2 and
Python 3.
Why Funding Is Needed
---------------------
The funding will service two high-level requirements:
- If Python 3 is to be maximally useful to web developers, they need
tools that make working with WSGI easier. Working with "bare" WSGI
as an application development platform is error-prone and difficult.
The lack of a shared request/response abstraction on Python 3 leads
to duplication of effort and dissatisfaction with Python as a web
application development platform. Existing Python 3-compatible web
frameworks (Bottle, CherryPy) each implement their own
request/response abstractions which are not divisible (or at least
not separated) from their respective framework. Continuation of
this pattern will lead to further fragmentation of the Python web
development community as time goes on.
- If Python 3 is to be relevant to existing Python WebOb-based web
framework authors, a Python 3-compatible WebOb must be made
available to them. Existing web frameworks which depend on WebOb
cannot be ported to Python 3 without being able to depend upon a
Python 3-compatible WebOb implementation.
WebOb currently does not run on Python 3. Creating a version that
will run on Python 3 will satisfy the above requirements.
The person or group who successfully ports WebOb to Python 3 must have
the following skills: a deep understanding of HTTP and related
specifications, a deep understanding of WSGI, mastery of Python 2,
mastery of Python 3, mastery of testing tools, and a deep
understanding of WebOb design goals. There aren't very many people
with the prior qualifications who aren't already engaged writing
Python web applications full-time on Python 2. They can't use Python
3 without having a suitable toolchain available, and therefore they
don't see much benefit in using it. Making WebOb available on Python
3 is also a rather boring and unrewarding task. As a result, it has
been difficult to convince anyone to to take the time from their
schedule to provide an initial Python 3 port. It's a bit of a
chicken-or-egg scenario.
Funding a port of WebOb to Python 3 will break the chicken-or-egg
cycle, and make Python 3 a viable platform to the people who use
WebOb, and, eventually, people who use frameworks based on WebOb.
I submit this proposal mainly on the advice (via Twitter) of Jesse
Noller, who replied to my public reports of trying to port WebOb on my
own time with a suggestion that I apply for a grant
(http://twitter.com/#!/chrismcdonough/status/108292018182172673).
Strategy
--------
The strategy will be to have a single version of WebOb that will run
on both Python 2 and Python 3 until such time as Python 2
compatibility can be retired.
A Python 2-compatible version of WebOb must continue to exist for the
foreseeable future, as a large number of existing projects depend upon
the combination of Python 2 and WebOb. Both bugfix and feature
releases of WebOb will need to be made indefinitely for the Python 2
platform.
A fork of WebOb to make it run on Python 3 that abandons Python 2 is
untenable. There is no one willing maintain a Python 3-only version
of WebOb. Likewise, there is no one willing to synchronize changes
made to separate Python 2 and Python 3 versions of WebOb, and having
the two versions fall out of sync is undesirable for forward and
backward compatibility purposes.
Therefore, we propose to use a single codebase to support both Python
2 and Python 3. This codebase will work under CPython 2.5, 2.6, 2.7,
and 3.2, PyPy 1.5 and 1.6, and Jython 2.5.2.
Since WebOb is very heavily based around I/O (taking data from the web
and presenting it to the developer in a form he can work with),
automated porting strategies such as 2to3 don't help very much; lots
of code that does encoding and decoding between bytes and text needs
to be changed "by hand". Likewise, details of the new WSGI
specification detailed in PEP 3333 require attention to detail that
fall outside the scope of any automated translation. 2to3 automated
translation is also costly in terms of library installation wall-time.
Therefore, the WebOb codebase will "straddle" Python 2 and Python 3
using a shared subset of both languages and will not rely on any
automated translation tools.
WebOb has no non-stdlib dependencies that aren't used solely for
testing purposes. We will rewrite any existing tests that have
Python-2-only testing requirements to use Python 3-compatible
analogues.
Prior Work
----------
In March of 2011, the Pylons Project sprinted at PyCon with the goal
of providing 100% unit test statement coverage to WebOb. This goal
was reached and laid the groundwork for being able to port to Python 3
by making it possible to have some level of confidence that Python
3-compatibility-related changes do not break existing code.
In the summer of 2011, Google sponsored student Joe Dallago to help
port WebOb to Python 3 via their Google Summer of Code program. This
project was mentored by people from the Pylons Project. Joe did the
best he could, but the task was too complex to be completed
successfully within the timeframe of GSOC. However, his work at
https://bitbucket.org/jayd3e/webob-1.1-py3 will serve as a basis (or
at least as a guideline and reference) to a funded porting effort.
Schedule and Budget
-------------------
The amount of work required to perform the port as described above in
"Strategy" is estimated to be 80 hours. The work can be broken down
like this:
- Remove all existing deprecated methods which promise results as
binary data (``str_GET``, ``str_POST``, ``str_params``).
- Redesign existing "MultiDict" strategy to not encode/decode at
lookup time, but instead to decode to text at request.POST /
request.GET / request.params dictionary creation time.
- Redesign existing cookie encoding/decoding strategy to work in terms
of text instead of binary.
- Redesign existing request and response descriptor implementations
which take environ data and return binary data to expect and return
text instead.
- Get all existing WebOb tests passing on Python 3.
- Functionally test on Python 2 and 3 under wsgiref, the CherryPy WSGI
server, and mod_wsgi. On Python 2, functionally test under
paste.httpserver.
- Run "in the wild" middleware, frameworks, and applications under the
new codebase on Python 2, to ensure we haven't broken anything (use
of non-API-but-probably-should-be-API methods/classes/functions).
- Ensure 100% test statement coverage on both Python 2 and Python 3.
- Update documentation to detail differences between usage of WebOb on
Python 2 and Python 3; also note any behavior differences from prior
releases.
- Work with Sergey Schetenin to merge the code into the WebOb mainline
repository. We have already discussed the need to support Python 3,
so this will be a code review process, not a process of defending
the need for a Python 3-compatible version.
Coding work will be done on a public Bitbucket fork of the main WebOb
repository.
The work will begin on October 1 and will finish on October 31. I am
requesting a total of of $3000.00 for this effort, to be paid on or
before November 30.
Risks
-----
There is not much "big design" thought required to do this work; just
lots of "little design" requirements. Most of the truly hard legwork
has already been done by the Python core team (wsgiref), by Phillip
Eby in PEP 3333, by Joe Dallago in his GSOC efforts, and by Graham
Dumpleton and Bob Brewer in mod_wsgi and the CherryPy WSGI servers,
respectively. There is little risk of getting the design wrong, as
the WebOb design already anticipates most requirements of encoding and
decoding web data from binary to text.
The only real risk is to existing users of WebOb under Python 2. We
need to make sure it continues to work for them and isn't backwards
incompatible for indefensible reasons. I'll attempt to make sure of
this by running "in the wild" middleware under the new implementation,
and fixing things as appropriate.
Qualifications
--------------
I am the primary author of the Pyramid web framework, which uses WebOb
as a base. I have contributed regularly to WebOb, and I have
collaborated with its maintainers many times. I have served as a
Google Summer of Code mentor this year to a student working on WebOb.
I am the original author of Python PEP 444 ("Web3", a WSGI replacement
protocol), although I am no longer its maintainer. I have been
developing Python web applications for 11 years.
Conclusion
----------
For a relatively small amount of funding, I believe we can make Python
3 a more pleasant web development platform, recognizable to people who
have a history developing web applications under Python 2. New users
will also benefit by having a reasonable request/response abstraction
rather than dealing with "bare" WSGI or being tied to a particular
framework implementation when they use Python 3.

I'm happy to report that this proposal was accepted by the PSF, and that
we've now released a Python 3 compatible
version of WebOb to PyPI .

Functionally test on Python 2 and 3 under wsgiref, the CherryPy WSGI
server, and mod_wsgi. On Python 2, functionally test under
paste.httpserver.

DONE: the codebase works under wsgiref and mod_wsgi; the CherryPy WSGI
server is not yet PEP 3333 compliant so it could not be tested.

Run "in the wild" middleware, frameworks, and applications under the
new codebase on Python 2, to ensure we haven't broken anything (use
of non-API-but-probably-should-be-API methods/classes/functions).

DONE: The Pyramid, Pylons, Restish, and ToscaWidgets 2 test suites were ensured
to run using the new code (not all tests passed but stakeholders with failing tests
were advised of potential backwards incompatibilies).

Ensure 100% test statement coverage on both Python 2 and Python 3.

DONE: We have 100% test coverage under Python2, but not under Python 3.
Some of the code executes solely on Python 2 and some solely on Python 3
and no existing coverage tool allows us to conditionalize coverage like
this.

Update documentation to detail differences between usage of WebOb on
Python 2 and Python 3; also note any behavior differences from prior
releases.

DONE: WebOb docs adjusted thanks to Sergey.

Work with Sergey Schetenin to merge the code into the WebOb mainline
repository. We have already discussed the need to support Python 3,
so this will be a code review process, not a process of defending
the need for a Python 3-compatible version.