NetWorkSpaces for Python

NetWorkSpaces (NWS) is a new way to write parallel
programs. It allows you to take advantage of multicore and
multiprocessors computers, as well as clusters, using
scripting languages such as Python, R, and Matlab. With
NetWorkSpaces for Python, you can execute Python functions and
methods in parallel using methods very much like the standard
Python map function. In some cases, you may be able to
parallelize your program in minutes, rather than months.

It looks pretty simple, and if you're already familiar with
the imap function in the standard itertools
module, you'll have no trouble with NetWorkSpaces. And yet,
it's doing something extremely useful and powerful. You can
easily submit millions of tasks that will be executed in
parallel, with the results returned to you for processing as
they arrive. NetWorkSpaces leverages Python iterators to
perform that trick, and by using other functions in the
itertools module, you can do in a few lines what would
take thousands of lines of C and MPI code to do.

The example above executes three local workers, which is
great if you have a four core processor, but if you're using a
cluster, all you have to do is construct your Sleigh as
follows:

Now, your Sleigh will start workers on the specified
list of machines using ssh. Submitting tasks and processing
results remains the same. This makes it easy to develop
your programs on your laptop, and deploy it later on a
cluster.

NetWorkSpaces is also pretty easy to install, being written
in pure Python, and installed using the standard Python Distutils.
However, for Windows users, we now have an "all-in-one"
NetWorkSpaces installer, available at
http://www.lindaspaces.com/downloads/NWSSetup-1.5.1.exe.

So give it a try, and find out how easy it is make your
Python programs run faster.