The foundations of dask.distributed provide abstractions to hide some
complexity of concurrent network programming (#2). These abstractions ease the
construction of sophisticated parallel systems (#1) in a safer environment.
However, as with all layered abstractions, ours has flaws. Critical feedback
is welcome.

Concurrency with Tornado Coroutines

Worker and Scheduler nodes operate concurrently. They serve several overlapping
requests and perform several overlapping computations at the same time without
blocking. There are several approaches for concurrent programming, we've
chosen to use Tornado for the following reasons:

Endpoint-to-endpoint Communication

The various distributed endpoints (Client, Scheduler, Worker) communicate
by sending each other arbitrary Python objects. Encoding, sending and then
decoding those objects is the job of the :ref:`communication layer <communications>`.

Ancillary services such as a Bokeh-based Web interface, however, have their
own implementation and semantics.

Protocol Handling

While the abstract communication layer can transfer arbitrary Python
objects (as long as they are serializable), participants in a distributed
cluster concretely obey the distributed :ref:`protocol`, which specifies
request-response semantics using a well-defined message format.

Dedicated infrastructure in distributed handles the various aspects
of the protocol, such as dispatching the various operations supported by
an endpoint.

Servers

Worker, Scheduler, and Nanny objects all inherit from a Server class.

.. autoclass:: distributed.core.Server

RPC

To interact with remote servers we typically use rpc objects which
expose a familiar method call interface to invoke remote operations.

.. autoclass:: distributed.core.rpc

Examples

Here is a small example using distributed.core to create and interact with a
custom server.

Client Side with rpc

RPC provides a more pythonic interface. It also provides other benefits, such
as using multiple streams in concurrent cases. Most distributed code uses
rpc. The exception is when we need to perform multiple reads or writes, as
with the stream data case above.