Thursday, December 9, 2010

HipChat has been changing what it means to communicate in teams of any size and taking the world by storm over the past 12 months. What you may not know is that Twisted is at the core of HipChat's highly successful service:

HipChat is a hosted group chat and IM service for businesses which uses the XMPP protocol behind the scenes. We use Twisted to power our fully clustered chat server and run many important job queue workers.

Bottom line: we built and launched a successful business on Twisted & Python in 6 months with no prior knowledge of either. It's a very productive environment with great performance and maintainability. What more could you want?

– Garret Heaton, Co-founder, HipChat

To learn more about how HipChat uses Twisted check out the success story!

Monday, November 29, 2010

Twisted 10.2.0, the third Twisted release of 2010, has emerged from the mysterious depths of Twisted Matrix Labs, as so many releases before it. Survivors of the release process - what few there were of them - have been heard to claim that this version is "awesome", "even more robust", "fun-sized" and "oven fresh".

Crossing several things that shouldn't ought to be, including the streams and the rubicon, I have assumed the triple responsibilities of feature author, project leader, and release manager for 10.2: with this dark and terrible power - a power which no man ought to wield alone - I have wrought a release which contains many exciting new features, including:

Look upon our Twisted, ye mighty, and make your network applications event-driven: get it now, from:
<http://twistedmatrix.com/>
... or simply install the 'Twisted' package from PyPI.

Many thanks to Christopher Armstrong, for his work on release-automation tools that made this possible; to Jonathan Lange, for thoroughly documenting the process and thereby making my ascent to the throne of release manager possible, and to Jean-Paul Calderone for his tireless maintenance of our build and test infrastructure as well as his help with the release.

Most of all, thanks to everyone who contributed a patch, reported a bug or reviewed a ticket for 10.2. Not including those already thanked, there are 41 of you, so it would be a bit tedious to go through everyone, but you know who you are and we absolutely couldn't do it without you! Thanks a ton!

Saturday, August 7, 2010

Fluidinfo is Twisted's latest sponsor! We asked if they could share some of the joys of sponsorship with us, and they had this to say about it:

At Fluidinfo, we're heavy users of Twisted. All of our infrastructure depends on it. FluidDB, our social database, is entirely built on Twisted, and we've released several core parts of it as open source: txAMQP, txRDQ and txThrift. Not only that, but we have contributed to Twisted both with code and (albeit small) personal donations.

It's not just that sponsoring was the fair the thing to do, it has also produced tremendous results in a framework crucial to our business. When you donate to the TSF it's simple math that more bugs get fixed, but also the quality of the entire Twisted project is enhanced. And when that happens, all of our products are enhanced automatically without us writing one line of code. So sponsoring is not just an act of generosity, it's an investment in Fluidinfo. Using and sponsoring Twisted has been an indispensable "force multiplier" for a growing start-up like us.

You can find more details about these tickets in the Twisted issue tracker. To look up a ticket, visit http://twistedmatrix.com/trac/ticket/ticketnumber. For example, http://twistedmatrix.com/trac/ticket/4487

Many of these which are still open are now up for review and should be closed over the next few weeks as other developers have a chance to take a look and provide feedback.

A careful reading of this list reveals a few tickets related to our attempts to convert Twisted's documentation from Lore to Sphinx, an undertaking you already know about if you follow the Twisted mailing list. Kevin Horn has spearheaded this project and made great progress towards the goal. Even though the conversion isn't complete yet, there's already some really great looking results.

Thursday, May 27, 2010

Think Twisted is only good for Internet-facing servers? Even then, it's not like Python itself that touches every part of your day in some way, right? Well my friend, you'd be delightfully wrong.

When you watch a movie crediting Lucasfilm for special effects, Twisted could be behind the scenes. Since 2004, Lucasfilm has used Twisted as part of its rendering operations to proxy database connectivity in the render farm (and much more), 24/7 x 365. That's right my friends, Twisted is not only the engine of your Internet...it might have a hand in your entertainment too.

To read more check out the great Success Story from Dave Peticolas from Lucasfilm: Twisted at Lucasfilm

Thursday, May 20, 2010

With the multitude of distributed version control systems now available, it's getting pretty hard for any open source project to get by on a centralized system. Distributed systems like bzr, mercurial, or git give equal tools to all contributors, not just a select few trusted to vette changes for inclusion in an official branch. Forcing new contributors to work with diff and patch is just plain mean!

So, while Twisted's source is still hosted in subversion, and likely will be for some time to come (due to various toolchain requirements), I'm happy to announce that there is now a git mirror to complement the existing bzr mirror.

With these mirrors available, developers who don't have commit access to the subversion server can still work with Twisted using modern revision control tools - branching, merging, history, and all the rest.

The workflow for merging changes based on the bzr or git mirror back into Twisted trunk isn't yet completely worked out. Likely it won't be until people start using these and trying to push changes back to us. So if you're a fan of one of these two systems, please, start using these mirrors and help us figure out the best way to integrate things back into subversion.

Sunday, April 25, 2010

As anyone who's contributed to Twisted knows, we currently have a very high standard for code changes and additions. For a long time, we've relied mainly on the standard library trace module. The trace module is quite serviceable, but recently some other options have come along which are much better. In particular, coverage.py seems to provide enough additional features that it's actually worth integrating into Twisted's development process. For example, it has some basic branch coverage features (trace will show you what lines were executed; coverage.py will show you this, plus tell you whether both possible outcomes of your "if" statement were taken), and can generate fairly nice HTML reports.

So, as a first step, I've set up a coverage.py builder on Twisted's BuildBot. This means that code coverage reports are part of Twisted's continuous integration system now: for every trunk revision, new coverage reports will be generated. At the moment, these reports end up jumbled up in the builds directory along with a lot of other stuff. For example, here's the latest report as of this post. Something I still want to do is clean up how these reports are organized so they're easier to find and compare.

As you can see from this report, Twisted has 86% line coverage and 67% branch coverage (which I had to compute myself from the totals at the bottom of the page - one of a few coverage.py nits that I hope will be fixed soon :). coverage.py decides this makes overall coverage 83%, though I'm not sure how exactly it arrives at that number.

That's pretty good, but there's plenty of room for improvement. I hope these reports will become an easy way for people to find areas of Twisted which need more coverage and motivate them to contribute new tests.

Oh, and since coverage.py only reports on Python coverage, I've also set up a gcov/lcov builder which uploads its coverage reports for execution of the C code in Twisted to a similar location.

Tuesday, March 30, 2010

We've added a great success story for PowerCard, a loyalty rewards program for local restaurants. One of the terrific things about the PowerCard story is the use of Twisted for both back-end and client-facing systems. It's a great reminder that Twisted is as useful solving problems on Windows platforms as it is on the Linux/UNIX-based systems many of us code for.

New Library ListingLooking to use Twisted with new technologies like RabbitMQ (txAMQP) and Redis (txRedis)? Browse the updated libraries listing. We bet you'll find what you're looking for.

New Twisted Ambassador

And why do we have these new and updated listings for 3rd-party code? It's because we have a new guy who has volunteered to maintain and update the listings!

Jason J. W. Williams is Twisted's new ambassador to 3rd-party projects of all kinds.Primarily this responsibility involves keeping the "Success Stories" and "Projects Using Twisted" pages up to date, and occasionally writing small updates (like this one) for the news feed.

Please contact success at twisted matrix dot com if you have a success story about how you used Twisted, a pointer to some hitherto unknown code that uses Twisted, or for more information. And watch this space for more updates!

Monday, March 22, 2010

Yesterday's Twisted sprint at the Flumotion office was quite a success. There were six of us there, and many first time Twisted contributors.

Jan Urbański worked on fixing a couple IMAP4 clientsearch bugs. He also spent some time using psycopg's asynchronous interface to create a very Twisted-friendly postgres API, with methods that return Deferreds, and all without using any threads.

And I walked around and answered questions about Twisted development and made suggestions about implementation approaches to try out. I also reviewed and merged the AMP documentation fix, and also worked on another IMAP4 ticket, improving test coverage for the client's SELECT support.

Many thanks to Jean-Paul Calderone and Chris Armstrong, whose work on release automation tools and answers to numerous questions made this possible. Thanks also to the supporters of the Twisted Software Foundation and to the many contributors for this release.

Tuesday, March 2, 2010

Residents of the once-great city of Atlanta are only now beginning to piece their lives back together after the event that Glyph Lefkowitz described as "the best Twisted sprint ever".

Quite apart from the unprecedented levels of wanton destruction, Twisted's PyCon 2010 sprint was also better attended than any previous sprints, which have laid waste to the outskirts of Boston, inner-city Dallas and regions of Australia.

At its peak, there were eighteen people seated around the two tables where the sprint took place. People even joined in remotely, working with the Atlanta team through #twisted on freenode.

Although buried deep within a secret bunker in Atlanta, the sprint still managed to attract a fair number of new contributors and Twisted users who wanted to help out.

Attendees all had immense fun and expressed a desire to have another sprint soon. One observed that it had been far too long since giant mecha robot warriors descended on Paris.

Twisted prevails,
jml

More simply, we had a great time at the sprint and got a ridiculous amount of stuff done.

We fixed many, many bugs, including all of the known regressions since the 9.0 release. Once we fixed the regressions, we started work on getting the first 10.0 pre-release out the door.

The pre-release probably would not have been possible without the sprint since it allowed the new release manager ready access to Jean-Paul, who has quite a lot of knowledge stuck in his head. One result of this is a new release process document, which is being developed alongside the actual 10.0 release.

We also reached an in-principle agreement to aim for a release every three months. This means that although hardly any of the good work from the sprint is in the 10.0 release, you won't have to wait long for it to appear in 10.1. Hopefully. I plan write up how this will work as soon as the 10.0 release is done.

The sprint proved to be a perfect opportunity to talk about some of the more gutly bits of Twisted, including Deferred cancellation (see #990) and endpoints (see #1442). The former is done, and the latter has made good progress. Stay tuned.

Thomas Hervé did a lot of work on upgrading our Trac instance, which was getting kind of old and currently crashes all of the time.

Maciej Fijalkowski was at the sprint working on getting PyPy benchmarks for Twisted and on making PyPy work faster with Twisted. Personally, it's really exciting to see PyPy live up to its promise of a faster Python, particularly with a big codebase like Twisted.