Back in August, I wrote
twoposts
about an experimental framework for distributed functional programming
called Girder. The idea, in summary,
was to make distributed code look as much as possible like ordinary
Clojure, as opposed to structuring it explicitly around message passing
(as in the actor model) or data flows (as in map/reduce, storm, et al).
As I say, it was an experiment, but it was also something a cri de coeur
(a mini-manifesto, if you will)
against extraneous impingements on my code. Anyway,
it sounds interesting, go back and read the posts, but you don't need to …

In part 1 and part2 of this almost
unbearably exciting
series, I outlined the concept of distributing purely functional programs and went through some implementation
details of Girder.

So far, however, I've only asserted that it works, so today I want to start
explaining how I used Amazon Web Services to test pretty cheaply on gobs of
machines.

The art of AWS wrangling was somewhat new to me, and I went through a few iterations
before deciding on the right mix of tools. Needless to say, the right mix ended up being
fairly Clojure heavy, which helps to smooth out …

Update 2015-01-12

The algorithm as it exists in HEAD is somewhat different from the below,
in ways that I'll describe (eventually) in an another post. In some ways, it's
closer to Fork-Join, but with important differences to support reentrancy,
share results of duplicate requests and adjust for the costs of distribution.

Recap of recap

In a previous post, I introduced a
framework called Girder (the code is on
github), which aims to facilitate
Plain Old Functional Programming on distributed systems. By POFP,
I mean code that, as much as possible, consists of normal looking
calls to normal looking functions, the …

Update 2015-01-12

The algorithm as it exists in HEAD is somewhat different from the below,
in ways that I'll describe (eventually) in an another post. In some ways, it's
closer to Fork-Join, but with important differences to support reentrancy,
share results of duplicate requests and adjust for the costs of distribution.

OSS and Commercial Grids

Grid computing has
always suffered the reputation of a buzzword that one suspects might
not actually mean anything, but it has become especially ill-defined
with the rise of open-source distributed computation frameworks like
Spark, Storm
and Grampa Hadoop. These extensively documented
systems don't need much …