In order to better meet the needs of partners, Racemi needed to
build a private web service to facilitate tighter integration
between our applications and theirs. After researching the state of
SOAP development in Python, we were able to find a set of tools
that met our needs quite well. In this article, we will describe
the criteria we used to evaluate the available tools and the
process we followed to decide which library was right for us.

Racemi’s product, DynaCenter, is a server provisioning and data center
management software suite focusing on large private installations
where automation is key for our end-users. Because we are a small
company, our business model is organized around partnering with larger
companies in the same industry and acting as an OEM. Those partners
typically provide their own user interface, and drive DynaCenter’s
capture and provision services through our API.

Many of our partners’ automation and workflow management systems are
designed to call scripts or external programs, so the first version of
our API was implemented as a series of command line programs.
However, we are increasingly seeing a desire for more seamless
integration through web service APIs. Since most of our partners are
Java shops, in their minds the term web service is synonymous withSOAP (Simple Object Access Protocol), an HTTP and XML-based protocol
for communicating between applications. Since Python’s standard
library does not include support for SOAP, we knew we would need to
research third-party libraries to find one suitable for creating a web
service interface to DynaCenter. Our first step was to develop a set
of minimum requirements.

Basic Requirements

DynaCenter is designed with several discrete layers that communicate
with each other as needed. The command line programs that comprise the
existing OEM API communicate with internal services running in daemons
on a central control server or on the managed systems. This layered
approach separates the exposed interface from the implementation
details, allowing us to change the implementation but maintain a
consistent API for use by partners. All of the real work for capturing
and provisioning server images is implemented inside the DynaCenter
core engine, which is invoked by the existing command line programs.
The first requirement we established was that the new web service
layer had to be thin so we could reuse as much existing code as
possible, and avoid re-implementing any of the core engine
specifically for the web service.

This project was unique in that many of the features of full-stack web
frameworks would not be useful to meeting our short-term requirements.
We have our own ORM for accessing the DynaCenter’s database, so any
potential solution needed to be able to operate without a fully-
configured ORM component. In addition, we were not building a human
interface, so full-featured templating languages and integration with
Javascript toolkits were largely irrelevant to the project. On the
other hand, while we recognized that SOAP was a short-term requirement
from some of our partners, we did anticipate wanting to support other
protocols like JSON in the future without having to write a new
service completely from scratch.

We also knew that creating a polished product would require
comprehensive documentation.

We also knew that creating a polished product would require
comprehensive documentation. The WSDL (Web Service Definition
Language) file for the SOAP API, which is a formal machine-readable
declaration of what calls and data types an API supports, would be
helpful, but only as a reference. We planned to document the entire
API in a reference manual as well as with sample Java and Python code
bundled in a software development kit (SDK). We could write that
documentation manually, but integration with the documentation tools
was considered a bonus feature.

Finally, we needed support for complex data structures. Our data model
uses a fairly sophisticated representation of image meta-data,
including networking and storage requirements. DynaCenter also
maintains data about the peripherals in a server so that we can
reconfigure the contents of images as they are deployed to run under
new hardware configurations. This information is used as parameters
and return values throughout the API, so we needed to ensure that the
tool we chose would support data types beyond the simple built-ins
like strings and integers.

Meet the Candidates

Through our research, we were able to identify three viable candidate
solutions for building SOAP-based web services in Python.

The Zolera Soap Infrastucture (ZSI), is a part of the pywebsvcs
project. It provides complete server and client libraries for working
with SOAP. To use it, a developer writes the WSDL file (by hand or
using a WSDL editor), and then generates Python source for the client
and stubs for the server. The data structures defined in the WSDL file
are converted into Python classes that can be used in both client and
server code.

soaplib is a lightweight library from Optio Software. It also
supports only SOAP, but in contrast to ZSI it works by generating the
WSDL for your service based on your Python source code. soaplib is not
a full-stack solution, so it needs to be coupled with another
framework such as TurboGears or Pylons to create a service.

TGWebServices (TGWS) is a TurboGears-specific library written by
Kevin Dangoor and maintained by Christophe de Vienne. It provides a
special controller base class to act as the root of the service. It is
similar to soaplib in that it generates the WSDL for a service from
the source at runtime. In fact, we found a reference to the idea of
merging soaplib and TGWebServices, but that work seems to have stalled
out. One difference between the libraries is that TGWS also supports
JSON and “raw” XML messages for the same back-end code.

Now that we had the basic requirements identified and a few candidates
to test, we were able to create a list of evaluation criteria to help
us make our decision.

Installing

A primary concern was whether or not a tool could be installed and
made to work at all using any tutorial or guide from the
documentation. We used a clean virtualenv for each application and
used Python 2.6.2 for all tests. Initial evaluations were made under
Mac OS X 10.5 and eventually prototype servers were set up under
CentOS 4 so the rest of Racemi’s libraries could be used and the
service could work with real data.

The latest official release of ZSI (2.0-rc3) installed usingeasy_install, including all dependencies and C extensions. A newer
alpha release (2.1-a1) also installed correctly from a source archive
we downloaded manually. The sample code provided with the source
archive had us up and running a test server in a short time.

We were less successful using easy_install with TGWS because we
did not start out with TurboGears installed and the dependencies were
not configured to bring it in automatically. After modifying the
dependencies in the package by hand, we were able to install it and
configure a test server following the documentation. Once we overcame
that problem, we found that the official distribution of TGWS is only
compatible with TurboGears 1.0. By asking on the support mailing list,
we found patches to make it compatible with TurboGears 1.1 and were
then able to bring up a test server. Since TurboGears 2.x has moved
away from CherryPy, and TGWS uses features of CherryPy, we did not try
to use TurboGears 2.x.

We never did get soaplib to install. It depends on lxml, and
installation on both of our our test platforms failed with compilation
and link errors. At this point, soaplib was moved off of the list of
primary candidates. We kept it open as an option in case the other
tools did not pan out, but not being able to install it hurt our
ability to evaluate it completely.

Feature Completeness

Since we anticipated other web-related work, we also considered the
completeness of the stack. Although ZSI provides a full SOAP server,
it does not easily support other protocols. Since our only hard
requirement for protocols in the first version of the service was
SOAP, this limitation did not rule ZSI out immediately.

Because TGWS sits on top of TurboGears, we knew that if we eventually
wanted to create a UI for the service we could use the same stack. It
also supports JSON out of the box, so third-party JavaScript
developers could create their own UI as well.

Interoperability

Another concern was whether the tool would be inter-operable with a
wide variety of clients. We were especially interested in the Java
applications we expected our partners to be writing. Since we are
primarily a Python shop, we also wanted to be able to test the SOAP
API using Python libraries. In order to verify that both sets of
clients would work without issue, we constructed prototype servers
using each tool and tested them using SOAP clients in Python and Java
(using the Axis libraries).

Both ZSI and TGWS passed the compatibility tests we ran using both
client libraries. The only interoperability issue we came across was
with the SOAP faults generated by TGWS, which did not pass through the
strict XML parser used by the Java Axis libraries. We were able to
overcome this with a few modifications to TGWS (which we have
published for possible inclusion in a future version of TGWS).

Freshness

Our investigations showed that there had not been much recent
development of SOAP libraries in Python, even from the top contenders
we were evaluating. It wasn’t clear whether this was because the
existing tools were stable and declared complete, or if the Python
community has largely moved on to other protocols like JSON. To get a
sense of the “freshness” of each project, we looked for the last
commit to the source repository and also examined mailing list
archives for recent activity. We were especially interested in
responses from developers to requests for support.

The recent activity on the ZSI forums on Sourceforge seemed mostly to
be requests for help. The alpha release we used for one of the tests
was posted to the project site in November of 2007. There had been
more recent activity in the source tree, but we did not want to use an
unreleased package if we could avoid it.

The situation with TGWS was confusing at first because we found
several old sites. By following the chain of links from the oldest to
the newest, we found the most recent code in a BitBucket repository
being maintained by Christophe de Vienne. As mentioned earlier, the
project mailing list was responsive to questions about making TGWS
work with TurboGears 1.1, and pointed us towards a separate set of
patches that were not yet incorporated in the official release.

Documentation

As new users, we wanted to find good documentation for any tool we
selected. Having the source is useful for understanding how you’re
doing something wrong, but learning what to do in the first place
calls for separate instructions. All of the candidates provided enough
documentation for us to create a simple prototype server without too
much trouble.

Just as we expect to have documentation for third-party tools we use,
we need to provide API references and tutorials for the users of our
web service. We use Sphinx for all customer-facing documentation at
Racemi, since it allows us to manage the documentation source along
with our application code, and to build HTML and PDF versions of all
of our manuals. TGWS includes a Sphinx extension that adds directives
for generating documentation for web service controllers, so we could
integrate it with our existing build process easily. ZSI has no native
documentation features. We did consider building something to parse
the WSDL file and generate API docs from that, but the existing Sphinx
integration TGWS provided was a big bonus in our eyes.

Deployment Complexity

We evaluated the options for deploying all of the tools, including how
much the deployment could be automated and how flexible they were. We
decided to run our service behind an Apache proxy so we could encrypt
the traffic with SSL. All of the tools support the standard options
for doing this (mod_proxy, mod_python, and in some cases mod_wsdl)
so there was no clear winner for this criteria.

In addition to simple production deployment, we also needed an option
for running a server in “development” mode without requiring root
access or modifications to a bunch of system services. We found that
both ZSI and TGWS have good development server configurations, and
could be run directly out of a project source tree (in fact, that is
how the prototype servers were tested).

Packaging Complexity

As a packaged OEM product, DynaCenter is a small piece of a larger
software suite being deployed on servers outside of our control. It
needs to play well with others and be easy to install in the field.
Most installations are performed by trained integrators, but they are
not Python programmers and we don’t necessarily want to make them deal
with a lot of our implementation details. We definitely do not want
them downloading dependencies from the Internet, so we package our own
copy of Python and the libraries we use so that installation is
simpler and avoids version conflicts.

ZSI’s only external dependency are PyXML and zope.interface. We
were already packaging PyXML for other reasons, and zope.interface
was easy to add. TGWS depends on TurboGears, which is a collection of
many separate packages. This made re-distribution less convenient,
since we had to grab the sources for each component separately.
Fortunately, the complete list is documented clearly in the
installation script for TurboGears and we were able to distill it down
to the few essential pieces we would actually be using. Those packages
were then integrated with our existing processes so they could be
included in the Python package we build.

Licensing

Although Racemi does contribute to open source tools when possible,
DynaCenter is not itself open source. We therefore had to eliminate
from consideration any tool that required the use of a GNU Public
License. ZSI uses a BSD-like license, which matched our requirements.
The zope.interface package is licensed under the Zope Public
License, which is also BSD-like. TGWS and most of the TurboGears
components are licensed under a BSD or MIT license. The only component
that even mentioned GNU was SQLObject, which uses the LGPL. That would
have been acceptable, but since we have our own ORM and do not need
SQLObject, we decided to skip including it in our package entirely to
avoid any question.

Elegance

SOAP toolkits tend to fall in one of two camps: Those that generate
source from a WSDL file and those that generate a WSDL document from
source. We didn’t particularly care which solution we ended up with,
as long as we didn’t have to write both the WSDL and the source code.
We also wanted to avoid writing vast amounts of boilerplate code, if
possible. As you will see from the examples below, the tools that
generated the WSDL from Python source turned out to be a much more
elegant in the long run.

We also considered the helpfulness of the error messages as part of
evaluating the elegance and usability of the tools. With TGWS, most of
what we were writing was Python. Many of the initial errors we saw
were from the interpreter, and so the error types and descriptions
were familiar. Once those were eliminated, the errors we saw generated
by TGWS code were usually direct and clear, although they did not
always point at the parts of our source code where the problem could
be fixed.

In contrast, we found ZSI’s errors to be very obscure. It seemed many
were caused by a failure of the library to trap problems in the
underlying code, such as indexing into a None value as a tuple.
Even the errors that were generated explicitly by the ZSI code left us
scratching our heads on occasion. We continued evaluating both tools,
but by this time we were leaning towards TGWS and growing more
frustrated with ZSI.

Testing

Automated testing is especially important for a complex product like
DynaCenter, so being able to write tests for the new web service and
integrate them with our existing test suite was an important feature.
ZSI does not preclude writing automated tests, but does not come with
any obvious framework or features for supporting them, so we would
need to roll our own. TGWS takes advantage of TurboGears’ integration
with WebTest to let the developer write unit and integration tests in
Python without even needing to start a test daemon.

Performance

Once we established the ease of creating and testing services with
TGWS, we had basically made our choice for that library. However,
there was one last criteria to check: performance. Using the prototype
servers we had set up for experimenting with the tools, we took some
basic timing measurements by writing a SOAP client in Python to invoke
a service that returned a large data set (500 copies of a complex type
with several properties of different types). We measured the time it
took for the client to ask for the data and then parse it into usable
objects.

The data structure definition was the same for both services, and we
found no significant difference in the performance of the two SOAP
implementations. Interestingly, as the amount of data increased, the
JSON performance reached a 10x improvement over SOAP. Our hypothesis
for the performance difference is that there was less data to parse,
the parser was more efficient, and the objects being created in the
client were simpler because JSON does not try to instantiate user-
defined classes.

Prototyping with ZSI

We were somewhat familiar with ZSI because we had used it in the past
for building a client for interacting with the VMware Virtual Center
web service, so we started with ZSI as our first prototype. For both
prototypes, we implemented a simple echo service that returns as
output whatever it gets as input from the client. Listing 1 contains
the hand-crafted WSDL inputs for the ZSI version of this service.

To generate the client and server code from the WSDL, feed it into thewsdl2py program (included with ZSI). To add support for complex
types, add the -b option, but it isn’t required for this simple
example. wsdl2py will, in response, produce three files:

Listing 2

EchoServer_client.py is the code needed to build a client for theSimpleEcho web service.

Once generated, these files are not meant to be edited, because they
will be regenerated as part of a build process whenever the WSDL input
changes. The code in the files grows as more types and calls are added
to the service definition.

The implementation of the server goes in a separate file that imports
the generated code. In the example, the actual service is on lines
18–25 of Listing 5. The @soapmethod decorator defines the input
(an EchoRequest) and the output (an EchoResponse) for the call.
In the example, the implementation of soap_Echo() just fills in
the response value with the request value, and returns both the
request and the response. From there, ZSI takes care of building the
SOAP response and sending it back to the client.

Listing 6 includes a sample of how to use the ZSI client libraries to
access the servers from the client end. All that needs to be done is
to create a handle to the EchoServer web service, build anEchoRequest, send it off to the web service, and read the
response.

Prototyping with TGWebServices

To get started with TGWebServices, first create a TurboGears project
by running tg-adminquickstart which will prompt you to name the
new project and Python package, and then produce a directory structure
full of skeleton code. The directory names are based on the project
and package names chosen when running tg-admin. The top-level
directory contains sample configuration files and a script for
starting the server, and a subdirectory containing all the Python code
for the web service.

tg-admin will generate several Python files, but the important
file for defining the web service is controllers.py. Listing 7
shows the controllers.py file for our prototype echo server. The@wsexpose decorator on line 7 exposes the web service call and
defines the return type as a string. On line 8, @wsvalidate
defines the data types for each parameter. As with the ZSI example,
the actual implementation of the echo call just returns what is passed
in.

Listing 7

fromturbogearsimportcontrollers,expose,flashfromtgwebservices.controllersimportWebServicesRoot,wsexpose,wsvalidateclassEchoService(WebServicesRoot):"""EchoService web service definition"""@wsexpose(str)@wsvalidate(value=str)defecho(self,value):"Echo the input back to the caller."returnvalueclassRoot(controllers.RootController):"""The root controller of the application."""echo=EchoService('http://localhost:7000/echo/')

The auto-generated WSDL for the web service is accessible viahttp://<server>/echo/soap/api.wsdl. Listing 8 shows an example of
the WSDL generated by TGWS for the prototype EchoService. It includes
definitions of all types used in the API (lines 3–20), the request and
response message wrappers for each call (lines 21–26), as well as the
ports (lines 27–45) and a service definition (lines 46–51) pointing to
the server generating the WSDL document. Each port includes the
docstring from the method implementing it (line 29).

The tgwsdoc extension to Sphinx, distributed with TGWS, adds
several auto-documentation directives to make it easy to keep your
documentation in sync with your code. By using autotgwstype,autotgwscontroller, and autotgwsfunction, you can insert
definitions of the complex types, controllers, or individual API calls
in with the rest of your hand-written documentation. This was
especially useful for us because we already had a lot of text
explaining our existing command line interface. We were able to reuse
a lot of the material and document all three interfaces (command line,
SOAP, and JSON) with a single tool.

Implementation Considerations

Once we had chosen TGWS as our framework, we set about working on the
first implementation of our real service. This helped us uncover a few
small problems with our original “pure” design, and some details we
had not considered while prototyping.

For example, we wanted to make sure that our web service was not only
interoperable with Java clients, but also that the API made sense to a
Java developer. One tool that they might be using, the Java Axis
client, is built by feeding the WSDL file into a code generator to
produce source code for client classes. After we tried working with
the generated Java code, we adjusted our web service API to make it
more usable. For instance, Java doesn’t allow you to specify defaults
for method arguments, which caused problems with a couple of web
service calls that had a handful of required arguments along with many
optional keyword arguments. On the Java side, the caller would have to
pass in all 23 parameters to the call, most of them null placeholders
for the optional parameters. To address that, we moved all the
optional parameters to a separate “options” object that could be
populated and passed in for advanced operations.

There were other minor annoyances, such as the way a camelCase
naming convention resulted in nicer-looking Java code than theunder_scored naming convention typically used by Python
programmers. We ended up going with camelCase names for attributes
and methods of classes used in the public side of the web service.
After making these tweaks, it is not difficult to design an API with
TGWS that makes sense to both Java and Python client developers.

Testing in Java was another challenge for us to work out. We have a
large suite of Python tests driven by nose, and we ultimately were
able to automate the client-side Java testing using junit. We then
integrated the two suites by writing a single Python test to run all
of the junit tests in a separate process and parse the results
from the output.

In addition to developer tests, Racemi has a dedicated group of test
engineers who perform QA and acceptance tests before each new version
of DynaCenter is released. The QA team needed a client library to use
for testing the new web service. None of them are Java programmers, so
the Dev team took on the task of basic Java integration testing. But
for full-on regression testing and automation, QA needed something
lightweight and easy to get up and running with quickly. Suds fit
this bill quite nicely. It is a client-only SOAP interface for Python
that reads the WSDL file at runtime and provides client bindings for
the web service API. Armed with our WSDL and the Suds documentation,
our QA team was able to start building a client test harness almost
immediately.

Conclusions

At the beginning of our evaluation process, we knew there were a lot
of ways to compare the available tools. At first, we weren’t sure if
the code-from-WSDL model used by ZSI or the WSDL-from-code model used
by TGWebServices and soaplib would be easier to use. After creating
the simple echo service prototype with both tools, we found that
writing Python and generating the WSDL worked much better for us.
Because WSDL is an XML format primarily concerned with types, we found
it excessively verbose compared to the Python needed to back it up.
It felt much more natural to express our API with Python code and then
generate the description of it. Starting with the code also lead to
fewer situations where translating to WSDL produced errors, unlike
when we tried to manage the WSDL by hand.

Because WSDL is an XML format primarily concerned with types, we found
it excessively verbose compared to the Python needed to back it up.

As mentioned earlier, we ended up needing to patch TGWebServices to
make it work correctly with TurboGears 1.1. Those patches were
available on the Internet as separate downloads, but we decided to
“fork” the original Mercurial repository and create a new version
that included them directly. We have also added a few other
enhancements, such as the option of specifying which formats (JSON
and/or XML) to use when documenting sample types, and better SOAP
error message handling. We are working with Christophe de Vienne to
move those changes upstream.

TGWebServices stood out as the clear winner for our needs.

Aside from the ease of use benefits and technical merits of
TGWebServices, there were several bonus features that made it
appealing. The integration with Sphinx for generating documentation
meant that not only would we not have to write the reference guide as
a completely separate task, but it would never grow stale as code
(especially data structures) changed during the evolution of the
API. Getting the JSON for “free” was another big win for us because it
made testing easier and did not lock us in to a SOAP solution for all
of our partners. Couple that with the benefit of having the TurboGears
framework already in place for a possible web UI down the road, and
TGWebServices stood out as the clear winner for our needs.

On the cover this month we have a story about the development ofUrban Mediator, a tool for collaborative city planning created at the
University of Art and Design Helsinki. This story is another example of
how the variety of problems solved with Python is endless.

Michael Noll brings us a how-to for Writing a Personal Link
Recommendation Engine. By studying the data available through the
delicious.com API, Michael’s example app can find links that might be of
interest or related to links you already have bookmarked.

In Reactive Programming with Traited Python, Judah De Paula
discusses how Traits can be used to add explicit typing, reactive
programming, and fast user interface development to your application.

JC Cruz creates a simple text editor to show us how to use an MDI
interface in Multiple Documents on PyObjC.

In Mark Mruss’ Welcome to Python column this month we learn the
basics of working with file I/O.

Brandon Rhodes regales us with the story of how Python’s universal
newline mode gave him headaches at a recent client engagement and, more
importantly, how he solved the problem.

In this month’s installment of Pragmatic Testers, Grig Gheorghiu
covers using mock objects as a way of simulating components to simplify
test setup.

And finally, Steve Holden covers some of the many Python information
sources available online (and off), and suggests ways for you to help
improve those resources.

Editing a magazine like this every month is, not surprisingly, ahuge amount of work. With the evolution and expansion of my duties
at my day job (almost everyone at MTA is part-time, even me), I’ve found
less and less time available to dedicate to my editorial duties. It was
a tough decision, but rather than let the quality of the magazine suffer
from lack of attention I found someone to take over who will be able to
devote the time necessary to keep up our high standards. I am very
pleased to announce that, beginning with the March 2009 issue, Brandon
Rhodes will assume the duties of Editor in Chief.

I met Brandon through PyATL, the Atlanta Python user group, where
he frequently gives presentations and is now the lead organizer. After
convincing him to write a couple of articles for us, I invited him to be
an Associate Editor for the magazine. His work as an AE over the past
year made him stand out as an excellent candidate to replace me. We’ve
been working on the transition for a little while now, so I have no
doubt it will go smoothly. I am looking forward to watching PyMag
continue to thrive through Brandon’s skill and leadership.

This past year has been a great opportunity and a lot of fun. I want
to thank MTA, and all the feature authors, columnists, editors,
reviewers, as well as Arbi, Marco, and everyone else who worked with me.
I’m proud of the work we’ve all done together!

I also want to thank our readers and the larger Python Community for
their support. In the spirit of open source we have been welcomed warmly
and enthusiastically over the past year, even though the magazine was a
new enterprise and MTA was not widely known among Pythonistas. Your
continued support (and writing) will help make Python Magazine a
valuable resource for all of us, so keep those ideas and article
submissions coming.

As for me, I plan to continue to contribute in smaller ways, such as
an occasional article or column. I also have a long list of other
projects that I have put aside for the past year, and it is time to pick
up one or two of those and see what I can make of them. If you’re
interested, keep an eye on this blog for updates.

The Python writing community has never been stronger. From blogs tomagazines to a slew of recent releases from O’Reilly, Packt,Manning, Apress, and other publishers there a more opportunities
than ever before for anyone wanting to write about Python in some
form.

Of course, authoring is only one aspect of producing high quality
professional writing. As an editor, it’s no surprise that I think all
authors can benefit from having someone else read their work. Even an
informal review can help identify gaps in an explanation or problems
with flow, not to mention typos.

Writers, reviewers, and editors can all benefit from sharing tips,
offering advice on tricky problem passages, or just chatting about
ongoing projects. That list sounds just like the sorts of things we talk
about when discussing programming, doesn’t it?

I’d like to bring people together in person for a group discussion and
a chance to network. To kick things off, I’m working on organizing a
“meetup” as an open space session at PyCon in March. If you are a
blogger, published writer, reviewer, or editor – or aspire to be – I
want you to come and participate. No prior experience is necessary,
everyone is welcome.

Let’s connect people interested in doing technical reviews with
authors and editors. Let’s introduce potential writers to the editors
and publishers. Let’s learn from one another and improve our prose, the
same way we improve our code.

We’ll have to wait to schedule a precise time until the conference,
but right now I’m thinking about late Saturday morning or early
afternoon. If you are interested in attending, please post a comment so
I can gauge the size of the room we’ll need. If you have a preference
for the time, make sure to include that information. And if you have
ideas for things we should talk about, let’s hear them.

Update: Although I was thinking of “print” writing, other forms of
publishing like podcasting are welcome, too. We need more good Python
podcasts!

This month’s cover feature is Creating a Collection Manager with
Elixir, by Gaëtan de Menten. You have heard about SQLAlchemy, but never
found the time to actually try it? Or maybe you have heard about the
add-on to SQLAlchemy called Elixir, but don’t really know what it is
useful for. Here is your chance to see them in action.

JC Cruz continues his series of articles on using Python to create
applications for Mac OS X. In the last article, you learned how to edit
data with the NSTableView. This month, in Table Drag and Drop with
PyObjC, you will learn the mechanics behind drag-and-drop and how to
support it in a table.

If you’ve ever done extensive development in Django, you’ll know the
issues involved with changing schemas. In Django Migrations with
South, Andrew Godwin introduces South, a migrations library for Django,
and shows how it helps to solve many of the problems you face as your
Django project matures.

This month we are co-publishing a special-guest column by Ivo Jansch,
a regular contributor to our sister publication php|architect.Requirements Gathering in the Enterprise is a look at the processes
used in large enterprise software development.

Jesse Noller reflects on the new challenges of test engineering in the
software world in his column, And Now For Something Completely
Different: The Changing Face of Test Engineering.

Interacting with the file system can often be a frustrating test for
new programmers. This month Mark Mruss introduces the os module and some
of the more helpful functions it contains to help ease some of that
frustration in Welcome to Python: Working with Files and Directories.

Beginning this month, we are proud to present the column, Ask the
Pragmatic Testers, written by Titus Brown and Grig Gheorghiu. Both
Titus and Grig have a wealth of development experience, and we’re
excited to add them as regular contributors to the magazine. Many people
think that automated testing is about making sure your software works
right. They’re not wrong, but in Why test? It’s about complexity.
Titus and Grig remind us that there are deeper benefits – in
particular, managing the complexity of the build and development
environments.

This month Steve Holden ponders on the Zen of Python, sometimes with
less relevance that might be expected.

And in What does Python 3.0 mean for you?, I talk about the
reactions to the long-awaited release of Python 3.0, that came on
December 3, 2008. This is a big step forward in the evolution of Python,
as it provides an opportunity for the core developers to introduce
backwards-incompatible changes to the language and libraries and break
free of some past design decisions that have been deemed misguided or
short-sighted.

The long-awaited release of Python 3.0 came on December
3, 2008. This is a big step forward in the evolution of Python, as
it provides an opportunity for the core developers to introduce
backwards-incompatible changes to the language and libraries and
break free of some past design decisions that have been deemed
misguided or short-sighted.

The precise nature of the changes in Python 3.0 may not be clear to
anyone who hasn’t been following the progress of the release over the
last couple of years, and that seems to be causing consternation in
some parts of the community. There is no reason for alarm becausesupport for 2.x is not being dropped. While having separate,
somewhat incompatible, versions of the interpreter will cause some
confusion, it was always (and is still) the plan to support both
versions until enough code is ported to make dropping support for 2.x
realistic. That is expected to take several years.

It has been no secret that Guido and some of the other core developers
have wanted to make a variety of backwards-incompatible changes for
some time. Guido has been talking publicly about Python 3000 for 8
years. That’s at least two lifetimes in internet time! All of the
plans were discussed at length, in public, on the python-dev and
python-3000 mailing lists. PEPs were written and reviewed, and all of
the code has been available in the branch for some time. With the
actual release announcement, however, some people seem surprised to
find that they actually went through with the changes.

Failure to change means we stagnate, and stagnation leads to
obsolescence. The differences in 3.0 are necessary to keep
the language vibrant and growing. The benefits include changing the
methods on built-in types so they consistently return iterable “views”
instead of list objects; simplification of comparison overloading;
removing surprises by making integer division return floating point
results when appropriate; making Unicode the default for text strings;
function argument and return value type annotation; set and dictionary
comprehensions (like list comprehensions, but for sets and
dictionaries); and basic clean-up of dusty or crufty “corners” of the
language or standard library such as changing some module names to
follow the PEP 8 conventions.

Failure to change means we stagnate, and stagnation leads to
obsolescence.

By the time you read this, 3.0 will have been out for at least a
month and it is quite
possible that the 3.0.1 will have been released. Some developers have
begun to release updates to their libraries enabling them to work
under 3.0, and I’m sure more porting is ongoing. We have not reached
a critical mass, so most of the community will not be ready to move
up, but we are making progress. One important step, already being
implemented, is for platform vendors to include 3.x in their operating
system updates so that developers have easy access to it.

If you have libraries of your own to port, there are several resources
to help out. First, study the release notes and list of updates to
understand what has changed. Join the python-porting mailing
list and participate in discussions with other developers encountering
some of the same challenges you might see in your porting effort.
Check out 2to3, the tool
for applying some of the more common transformations to your Python
code automatically. And of course, Python Magazine will be covering
some of the differences between 2.x and 3.x over the course of the
next few months.

The move from 2.x to 3.x is going to take a considerable amount of
time for most users, and that is anticipated. While you should start
planning to upgrade, you should not rush out to move all of your
applications to 3.0 right away. There is every expectation, from the
core developers and most of the community, that the 2.x branch will be
supported for several years as we work on the migration.

New Columnists

As hinted in my last column, the new year brings changes for Python
Magazine, too. Beginning this month, we are proud to present the
column, Ask the Pragmatic Testers, written by Titus Brown and Grig
Gheorghiu. Many of you will recognize Titus and Grig from the large
and growing Python testing community. Both have a wealth of
development experience, and we’re excited to add them as regular
contributors to the magazine. They start their series by explaining
the benefits of testing as a way to manage the complexity of software
development. It’s a convincing argument for adding automated tests to
your project, no matter the size. Code has a way of growing over
time, so even small projects benefit by starting with automated tests
right from the beginning.

As the title of the column implies, Titus and Grig want to hear from
you. What sorts of testing are you doing, and what problems are you
encountering? What aspects of testing are most interesting to you?
Email your questions or suggestions for them to pragmatictesters at
pythonmagazine dot com.

If you have comments, criticisms, or ideas for topics we should cover
elsewhere in the magazine over the next year send them to me at doug
dot hellmann at pythonmagazine dot com.

Our cover story this month is Using the Cloud, in which Patrick
Altman shows you how to wield the power of globally distributed data
centers with infinite storage, immediate scalability, and no fixed costs
or large capital outlays using Python.

Michael Schneider brings us Design Patterns in Real Python
Projects, a survey of design patterns uses found “in the wild”.

JC Cruz continues his series of OS X programming tutorials inTabular Editing With PyObjC. This time, he covers editing data in a
table control.

Jesse Noller’s column, YAML Ain’t Markup Language, gives us
several good reasons to consider YAML as a substitute for XML,
especially in files meant to be read or written by humans.

Mark Mruss shows us how easy it can be to work with 3-D graphics using
VPython.

And this month Steve Holden takes a look at various developments in
and around the Python language, reports on an extensive discussion on
Python’s function call mechanism, and reminds you about Python
conferences.

Finally, I recount some of my experiences from PyWorks 2008 and look
ahead at how the magazine will grow in 2009.

The first annual PyWorks conference has just wrapped up, and it
was by all accounts a big success. We had a great time
socializing, learning about new tools, and catching up on the
progress made by established projects. The Atlanta weather was
unusually wet, but that didn’t stop some of us from heading
off site to attend the local Python user group meeting, too. All in
all, it was a fun and productive three days.

One of the unusual features of PyWorks
was that it was part of a combined conference. We shared space,
attendees, and session speakers with php|works, the PHP conference
also hosted by MTA. This allowed us to hold some interesting combined
sessions on topics useful to both PHP and Python developers, including
an introduction to git, tutorials and presentations on administering
databases, designing security into your applications, and many more.
It was good to have exposure to another coding culture, and I had an
informative conversation with a few of the PHP core developers about
how their development practices tend to differ from the BDFL model
used for Python core development.

Kevin Dangoor, founder of the TurboGears project, gave the opening
keynote, entitled Growing Your Community. He covered reasons for
attracting more users and contributors to your project, and tips to
help you accomplish it. Our very own Brandon Rhodes presented an
excellent Grok tutorial that has convinced me to give Zope another try
after a five-year hiatus. Another of our editors, Jesse Noller,
showed the work he has been doing to integrate the ‘’multiprocessing’’
module into Python’s standard library. Mark Ramm’s presentation on
WSGI middleware included somewhat creative visual aids (ask him about
it at PyCon). There were presentations on most of the major web
frameworks, desktop applications, the Sphinx documentation system,
artificial intelligence, systems administration, and more. The
breadth of topics presented meshed well with the varied interests of
the attendees.

PyWorks in November and PyCon in March are very different types of
conferences. For one thing, the attendance level at PyCon was much
larger. That can be a great benefit for attendees, because it means
there are more opportunities to meet people and a wider variety of
presentations to choose from over the course of the week. On the
other hand, while the numbers for PyWorks were lower, that made the
sessions more intimate. It was easier to ask questions and interact
with the presenters and other members of the audience. It felt a
little more like the open spaces session at PyCon.

Another way it was different, of course, was the fact that it was
combined with php|works. We had good feedback from conference-goers
about the format, along with a few suggestions to make it even better
next year. For example, one aspect of the Python community culture
that PHP developers aren’t familiar with is sprinting. There was a
TurboGears sprint in Atlanta the weekend after PyWorks, but we didn’t
know about it in time to really integrate it with the the conference.
Another thing we need to do is make sure we have a good cross-section
of Python programmers represented. Many projects were well
represented, especially Grok and TurboGears, but we can do better
about extending invitations into other sub-groups within the Python
community.

Looking Ahead

As a print publication, we always try to plan ahead to make sure we
have plenty of interesting articles and columns ready for upcoming
issues. We’ve been doing this ourselves for the past year, and so far
the feedback has been positive. Your input is always valuable,
though, if for no other reason than the small number of editors here
can’t match the entire community for sheer creativity. So we want you
to tell us, what exactly do you want to get out of your subscription?
What should we be doing that we aren’t? What topics or segments of
the community have we missed? How can we make the magazine even more
useful to you? Post a comment below and let me know what you think!

The cover story this month is Building E-commerce on Plone with
GetPaid by Horacio Durán. He walks through several scenarios for
configuring different types of sites that need to accept payments, from
charity donations to shipping physical goods.

Gloria Jacobs’ article, An Introduction to SQLAlchemy, uses a
straightforward intranet application to illustrate the power and
portability of SQLAlchemy.

In his regular column, Mark Mruss talks about the occasionally
contentious subject of using “slots” in Python classes. He shows how to
use them and covers the pros and cons to help you decide when they might
be useful in your own applications.

Rick Harding continues his bzr tutorial series by showing how to use
different workflows, depending on the style of development that makes
you most comfortable.

Steve Holden talks about ways you can contribute to Python, including
being active in the PyCon organization process and helping with the
language itself.

And I take a look back at the growth of the magazine over the last
year, since Python Magazine is now officially one year old!

At the PyWorks convention this week there were enough people talking
about interesting things I missed from not having a Twitter account that
I decided to abandon my resistance and sign up. I’m still setting up
the list of feeds I want to follow, and since people search is down I
expect that to take a while.