Swarm - Concurrency with Scala Continuations

By bagwell

Created 2009-09-28, 08:53

With Swarm[1], Ian Clarke demonstrates a novel approach to programming distributed systems using Scala's serializable delimited continuations. Ian, CEO of Uprizer Labs and developer of SenseArray[2], is possibly best known for designing and coordinating Freenet[3], a popular P2P distributed network storage application.

As recently announced[6], continuations will be included in Scala 2.8 as a compiler plug-in. Ian has innovatively applied this technology to create Swarm, a potentially ground breaking Open Source project that significantly eases the writing of programmes for loosely and tightly coupled multiple processors environments.

The delimited continuations feature of Scala allows you to save a function state and resume it later. Ian explains that, unlike other languages, the Scala implementation is unique in that the resulting object is portable; it can be transferred to another computer and resumed there. He then shows how this enables Swarm to move computations, in an efficient and potentially transparent way, between processors to operate on distributed data structures.

Ian demonstrates a proof of concept prototype and outlines data clustering concepts to be used. He finishes his presentation by outlining some of the challenges that lay ahead, the arrangement of the data for optimal efficiency, GC, multi-threading, class loader for a Global namespace and creating a web framework to use Swarm.

If you want to work on state-of-the-art Scala applications, this would be a fascinating project to join. See more about the project and source code here[7].

Jay McCarthy says "Our programs use the same features as non-scalable, continuation-based Web programs, so we do not sacrifice expressive power for performance. In particular, we allow continuation marks in Web programs. Our system uses 10 percent (or less) of the memory required by previous approaches."

Although Jay McCarthy used PLT Scheme, you could use the same technique just as well with the Scala continuations.