She also resolved #1247: ("kqreactor timeout error (traceback)"), and opened #5385, #5386, and #5387, while resolving #5060: ("Correct python requirement in twisted/topfiles/setup.py").

Not to mention all the tickets reviewed at recent Sprints which have since been resolved! (Though, to mention them anyway, #1247, #2115, #2447, #2498, #2507, and #5370)

Eternal High Score Overlord JP Calderone made a fair showing as well, finishing #5371 ("In the developer guide, forbid new global reactor unit tests"), and splitting #4671 ("sometimes I want to have some constant values that are part of a set") into more managable tickets:

One-word Twisted celebrity Glyph was also on hand to assist with the reviews of #3242, #5382, and #1918

With considerably more words in his name, Itamar Turner-Trauring started porting a Nevow branch with IRI support (internationalized URLs) to Twisted as per ticket #5388, as well as helping resolve #3037 with Jessica.

Finally, Glyph, Itamar, and Dave Sturgis conferred on #288, and the related cluster of web-resource-model issues it involves; It is hoped that this will eventually result in a new Web Server URL Traversal API, or WSURLTA, not to be confused with the nordic kraken-headed demigod of lore.

Tuesday, October 25, 2011

I'm happy to report that I've just completed another two weeks of sponsored Twisted development. As usual, the issue tracker directed most of my work. I spent time on new development aimed at resolving tickets and I reviewed changes proposed by other developers with that aim.

All told, 59 tickets got some attention that they would not have received without the sponsored development. The result was 36 closed tickets and 17 other tickets unblocked for other developers to resume work on. Over the coming days or weeks a few more tickets will probably be resolved as a result of this work, as developers (including myself) respond to review feedback.

This work is made possible by the sponsorship of individuals and organizations which have donated to the Twisted project, part of the Software Freedom Conservancy, a not-for-profit organization that helps promote, improve, and develop open source software. Thanks!

Apart from working to resolve tickets in the issue tracker, I also did some work on infrastructure. I:

Friday, October 21, 2011

After a brief rest, the Boston Twisted crew is back on the sprinting treadmill. Dave Sturgis came through for us once again for the venue, working with Smarterer to let us use their office space and eat their snacks last Saturday afternoon. We also took advantage of the Python Software Foundation's sprint funding program to buy all the sprinters dinner.

Chris (radix) (triumphantly returned from the inhospitable southwest) worked on adding a history-tracking mode to Deferreds to aid in debugging. This feature will let a developer see everything that has happened to a Deferred - what callbacks it has run, what values have passed through it, and what other Deferreds it has been associated with.

Jessica (jesstess) did a ton of follow-up on old tickets (1247211524472498250725132861), trying to learn if they are still valid and elicit further information from the original reporters. She also managed to close one which was no longer valid (but she also filed a new one for a documentation bug, in case you were worried we might run out of tickets).

Friday, September 23, 2011

Twisted has a somewhat extensively customized BuildBot master configuration which has evolved over the last ten years. We've kept it private only because it contains some passwords which it would be better not to share, but over the years we have wanted to make it public - because it might serve as a good example to others, because it would let more people hack on it and relieve us of some work, etc. At last this has changed: I spent a couple hours today filtering out the sensitive parts. As a result, you can now take a look at lp:twisted-buildbot-configuration. Add some features! Do some refactoring! Write some unit tests!

Wednesday, September 7, 2011

Twisted uses BuildBot for continuous integration - for testing every revision to ensure it passes the full test suite. BuildBot automatically runs the test suite on each version of Python and each operating system supported by Twisted. As of today, that's 22 different configurations, plus another 16 configurations where Twisted is known not to work completely but where we are workings towards full support.

This works very well for identifying regressions and for finding problems with new functionality on platforms a developer may not have considered. However, because Twisted prefers to keep trunk@HEAD completely working and the test suite fully passing on all platforms, if BuildBot identifies a problem with a trunk revision, we revert it completely. Then the problems with it can be addressed and another attempt made. This also works, but it has a couple shortcomings. Someone actually has to do the revert, which is a little bit of work. Also, someone might check out trunk@HEAD before the change is reverted, giving them a non-working version of Twisted. It's nice to be able to avoid both of these things.

The solution generally applied is to test the change before applying it to trunk. BuildBot can do this too, in a couple different ways. The way we use for Twisted development is to have BuildBot run the test suite on all those supported configurations on a branch of Twisted, rather than on trunk. Since the change is checked in to a branch, instead of to trunk, it avoids the needed to revert anything and it avoids offering broken code to people.

This approach involves a tiny custom tool to convince BuildBot to run the necessary builds. Normally BuildBot just watches for trunk commits and starts builds automatically. Since in this case we want the trunk commit to happen after we see that all the builds will succeed, we use force-builds.py to have BuildBot perform the builds before it sees the trunk commit.

For a long time, force-builds.py needed a hard-coded list of supported platforms on which to trigger builds. Recently I fixed this by having it retrieve that list from the BuildBot master, which already knows which platforms are supported. Publishing this information is done with a minor hack to master.cfg:

The resulting file is publicly available from the master's web server, so force-builds.py can download it on-demand and then force a build on each supported builder. When the developer sees all of these builds have succeeded, he or she can be more confident that there aren't going to be any surprises when the change is actually applied to trunk.

Wednesday, August 10, 2011

And far to the East where strife sired shadows A baelnorn.local did rise and approach the light. Out came illidan.local over once fair meadows, And in defeat left his people weeping for their plight. - kabuto.local's A Record

Jean-Paul and I had some hardware adventures a couple weekends ago. As these things usually are, it was harrowing - but perhaps less so than it might have been. By combining, in approximately equal parts, some unused components which have been ripening in the back of a closet in JP's apartment and some newly acquired materiels from the local vendor everyone (well, we, at least) loves to hate, and with only a few false starts involving incorrect memory channel configurations, Twisted now has the use of a somewhat hefty new computer, visible here in its new home, the undisclosed subterranean location that hosts all of Twisted's buildbot machines:

Don't ask my why it glows blue; it is my understanding that this is simply the fashion these days.

With enough cores and gigabytes of memory so that I have trouble counting them all, this machine should serve us well hosting many a build slave. It has already taken over the duties of several older slaves, including those of a couple Windows slaves, care of kvm (the machine itself is running Ubuntu Lucid (10.04) of course). The very nice network it is attached to means that remotely interacting with these is now only minimally unpleasant.

The machine is also translating PyPy (in a respectable 34 minutes) once a week so we can continue to test Twisted against the latest development versions of the PyPy runtime.

We've broken these tasks into bite-sized chunks, and you can find descriptions for them on the Project Jobs wiki page. Take a look, and please claim a task and how long you're up for doing it on the mailing list. If you have questions about any of them, don't hesitate to ask on the mailing list or IRC. You'll have lots of help getting set up.

Ticket openers, reviewers, and closers all get varying amounts of points. Note that a review banks you 1000 points, the most for a single ticket action, and we have a long review queue, so brush up on our patch review guidelines and dig in.

Tuesday, May 31, 2011

Greetings audience. Another month has gone by, and that means another excellent Twisted sprint is behind us. A couple weekends ago Dave, Ying, Itamar, Glyph, Asheesh, and myself gathered at the now traditional sprinting location. Quite a few things were worked on. Amongst the highlights were the following tickets:

#4813 - provide permissions accessor for filepath
#5053 - Add Gzip support to web client
#5088 - cfreactor cannot be imported
#5063 - When TLS transport is in writeBlockedOnRead=True mode, data is buffered in memory but producer APIs are not respected
#5095 - twisted.protocols.tls should flush all of the bio data, rather than just 2**15 bytes

Monday, May 16, 2011

Do you have a hankering to run your own multi-user text-based game server? If you do, and you love Twisted, Evennia is the project for you! Through the power of Twisted, Evennia has built a MUD-server platform that not only provides traditional telnet access but also lets you play via the web! As Evennia maintainer Griatch Art put it:

Whereas traditional MUD servers only speak telnet, Evennia additionally comes with its own browser-based client and allows for expanding with custom game protocols in any combination. ... All of this is straightforward to implement and maintain thanks to Twisted!

Evennia also combines Twisted with Django, providing a comprehensive real-world case study of melding these two popular systems. As Griatch puts it, Twisted and Django "work beautifully together".

To find out more about how Twisted was the perfect fit for this massively multi-player platform and why the Evennia team chose Twisted, check out the success story.

Many thanks to Glyph Lefkowitz and Jean-Paul Calderone for sanity-checking the pre-releases and release, and to the enthusiastic PyCon 2011 sprinters who annihilated dozens of tickets. Thanks to *everyone* who contributed tickets, patches, documentation, reviews, buildbots, feedback, and assistance to fellow users and developers leading up to this release. It is truly a group effort.

Monday, March 28, 2011

We held a small but productive and successful Twisted sprint, kindly hosted once again by tenth, and attended by glyph, exarkun, jesstess and lewq.

The aim of the sprint was to do the first pre-release of Twisted 11.0 - and we managed to resolve almost all of the blocking tickets, so a partial success. :-)

One of the two main themes of the work was the twisted.web templating code, which got tidied up and properly documented - there's a new tutorial (temporarily available at http://buildbot.twistedmatrix.com/builds/sphinx-html/291-15849/projects/web/howto/twisted-templates.html), which demonstrates this neat little templating engine in action. The other theme was completing and merging the file descriptor closing branch which exarkun and lewq worked on, bringing FreeBSD just a few test-fixes away from being a fully supported platform - and importantly stopped the FreeBSD builder hanging on testing trunk, which will help catch regressions.

#4854 - Replace the implementation of IReactorSSL with one based on twisted.protocols.tls #3690 - SerialPort never calls connectionLost(reason) #4969 - Add CONNECT support to HTTP client #4951 - links in twisted.web docs are broken

That's it until next time, enjoy Twisted 11.0 when it comes out! We still have the PyCon sprint to report on, watch for that soon.

Sunday, March 20, 2011

Last month, Twisted held another very successful sprint. A late write-up is better than no write-up, so here's what went on.

In attendance were Glyph, Tenth, Itamar, Mike Handverger, Luke, and myself (Jean-Paul). Remotely, Jonathan J. and Stephen Thorne helped out. Quite a few tickets were worked on, and various reviews were done as well. Here's a list (bold indicates now closed):

Monday, January 24, 2011

If you recently tried to donate money to the Twisted Software Foundation using PayPal, then you might find that your donation has gone "unclaimed". This is due to an unfortunate mixup in our PayPal address. If you have a donation in this state, you should cancel it; we have no way to claim it.

If you'd like to donate, use the PayPal form on http://twistedmatrix.com/. We've updated it so future donations will go to the right place. If you'd rather use Google Checkout, the form for donating with that service is also still available in the same place.

There was also a lively discussion about plans for the future support of Twisted on Windows. If you're interested in Windows networking, keep an eye out for an upcoming announcement on that topic!

Once again we were lucky enough to have tenth host the sprint, so in addition to shelter from the frozen terrorscape scant meters away, sprinters also enjoyed delicious fresh baked bread and other tasty morsels.

If you couldn't make it to this sprint, watch for an announcement of the next one sometime soon. Or join us at PyCon for as many as four days of Twisted hacking!