Twisted Python

At last month's PyCon in Washington D.C. several developers of the Twisted network application
framework gave presentations. (Papers are available on the Python
wiki.) In some ways, it was Twisted's big debut. Many attendees
have made positive remarked the presentations. Once obscured, Twisted
is stepping into the light.

The conference also gave Twisted's developers a chance to meet in
one place. The Twisted project was originally conceived on the
OpenProjects (now Freenode) IRC
channel, #python. Its core developers are spread all over the world.
While inspired by extreme programming--which among other things,
promotes programming in pairs--they rarely have an opportunity to do
work that closely.

I spoke with Chris Armstrong, a.k.a. radix, about both Twisted and
the conference. Armstrong lead a productive pre-conference
programming sprint at PyCon. 14 key Twisted developers were joined by
2 other developers to discuss and refine some important Twisted
modules. He told me that "it went great! We're incredibly productive
when together. It's not easy to do much pairing over the
internet--you don't get the same feedback."

I heard about Twisted a couple of years ago but never took a close
look at it. Helping design a distributed middleware application
recently, I decided to take a closer look at Twisted. It wasn't easy
for me to understand it. There was so much to Twisted I didn't think
I would need. I felt overwhelmed. I asked Twisted developer Chris
Armstrong, "what's the best way for someone to approach learning
Twisted?"

"I try to stress that they look at the parts they need for their
application," he told me. "Usually it's okay after that--depending on
what parts they need. Woven is still a moving target and sometimes
people have problems with that." Woven is the module used to create
web interfaces for networked applications. Armstrong tells me its
getting better. Some much needed documentation is on the way. "Start
with what you need. If you are writing a web application, don't look
at how HTTP is implemented, for example."

That said, I would have found Twisted easier if I had understood
asynchronous programming better. Rather than using multiple processes
or threads to handle multiple clients, Twisted uses an asynchronous,
event-driven model. I didn't begin to understand that until I started
to think of Twisted as being like a GUI application framework like Qt
or Gtk. GUI frameworks are also event driven application
frameworks.

What might also have helped was to spend a little time playing with
asyncore.py, Python's own built in asynchronous socket library.
Asyncore comes from Sam Rushing's Medusa, the framework used
by Zope. Sam has a low level tutorial on Medusa that would have
helped me immensely for general concepts. Twisted abstracts all of
this nicely. You don't have to understand how Twisted does things,
but I feel more comfortable understanding what's being abstracted.

Twisted originated as the underlying framework for an open source,
massively multi-player game called Twisted Reality. Armstrong tells
me when he got involved with the project in January of 2001, "I was in
High School and was bored. I was just getting into Python and Twisted
looked cool. I was interested in massively multi-player games.
Twisted started as a multi-player interactive fiction game, twisted
reality. Everything came out of that, the web server, the protocol
support." Eventually the split off Twisted Reality as a separate
project. But there is certainly an element of fun remaining in
Twisted, including module names that are puns on peanut butter and
jelly sandwiches.

Armstrong told me development of Twisted Reality continues. "When I
first got into the project, Twisted Reality development was going
strong, but since has faded off. We recently started a rewrite. It's
going slow, but hopefully that will change soon."

Many of the Twisted developers can still be found online daily at
the Freenode IRC #twisted and #python channels. Many users are there
as well. Twisted community support is strong, and it's easy to get
assistance either through IRC or the mailing list. For businesses
whose needs might exceed what the community can easily provide, a
couple of companies provide commerical support.

Twisted is the basis for several other open source projects that
exhibit the same friendliness and support. If you are looking for a
hot project to get involved with, Twisted is a great place to start.
Armstrong tells me working with Twisted has been a great place to
learn programming. "All my programming knoledege--I wouldn't say it's
self taught. Glyph (Lefkowitz) and Dash (Allen Short) have been great
mentors for me over the last few years. I have learned a ton from
Glyph and a lot of it though the Twisted project."

Twisted developers recently released version 1.0.4. This point
release includes several improvements as well as bug fixes, including
optimizations of Twisted's GUI integration tools. It also introduces
a new version of Jelly, Twisted.Spread's serialization module allowing
you to securely and safely pass objects over the network in a language
neutral way. NewJelly will eventually replace the existing Jelly, but
because it isn't backwards compatible, it's available as a separate
module. Twisted.Lore now provides support for generating presentation
slides.

Take a close look at Twisted before embarking on your next network
application project. Even if you have no need for the higher level
tools, you can take advantage of Twisted's handling of network
transports and protocols as well as it's asynchronous model to quickly
develop fast, robust, secure servers and clients.