Transducers are fundamental

Transducers
are being introduced in the next version of clojure. There are
a few posts about them already, but I found the notion of
transducers more elegant and natural than I have seen
described. They simplify a lot of disparate clojure concepts and
make it easier to write more performant code without any loss in
generality. They are quite beautiful, actually, and I believe the
concept is elemental, like function composition.

Definition

From the announcement: a transducer is a function that takes one reducing function and returns another.
That's very general and totally relies on understanding a reducing
function. A reducing function is a function that has the structure of what you'd
pass to reduce. It is a function which accepts an element of input and
a previous reduction, and returns a new reduction.

By Example

Making it more concrete by way of example, let's start with a simple reducing function:
This reducing function reduces a sequences of numbers into a map containing the count and sum of the numbers.

Now that we have a reducing function, let's transduce it,
using only what we know so far. Again, per the definition, this
means we'll need to return a function that's a reducing function
(one that accepts the previous reduction and a new element, and
returns the next reduction). Let's try to make a transducer that
will cause our mean-reducer to increment all of the inputs.

We've just implemented a crude transducer, and all it can do is increment.

Core Functions

One of the parts of the announcement of transducers was that
clojure's core functions (map, filter, take, etc) that normally
operate on sequences will gain a new arity that returns a transducer
when called with a single argument that's a function. The new code
for the 1-arity map looks much like what we already wrote
(ignore the other arities for now).

Now we can see in fact how crude our original transducer is, it can be
constructed trivially with map.

Neat, but if you stop and think for a minute, the above returns the same result as the non-transducers form:
Here, map is operating on a sequence like it did before.
Now we know what transducers are, but haven't quite had the epiphany yet.
So why is the transducer form different and more importantly, better?

This is the right question to ask! And will lead to understanding
the elegance of transducers. Don't proceed until you convince yourself
that the previous two examples generate the same result.

Decoupling From Sequences

When using the thrush, or ->>, form above, map
accepts an input sequence one by one, and runs inc over each of the
elements, generating a new sequence, which is then reduced by reduce.
This creation of logic via composition of
sequence-iterating functions complects the created logic
with the machinery of sequences.
This also means that unnecessary intermediate sequences are created
just to execute your logic.

If we could separate our logic from sequences, we could write more
performant code by avoiding incidental sequences. We could also use
the same logic across things that aren't necessarily sequences (eg:
clojure.core/reducers, channels).

Transducers let us do that by composing functions "inside" the
traversal of the input, instead of composing sequence-traversing
functions "outside" on the sequences themselves.

They are similar to traditional function composition
(eg: comp) in this way, but a comped function iterated over a sequence can only know
about a single element at a time, and doesn't have
state. Transducers can encapsulate state (eg: the take
transducer uses an atom to count elements), and can control
generation of output (eg: the filter transducer can decide
whether or not to pass a token to the passed-in reducing function,
whereas a composed function that's mapped needs to
generate one output per input).

Using Transducers

We've seen how you can create a transducer, and the advantages they
have, but how do you use them? Above we used our crude transducer
and our (map inc) transducer with reduce, but
because they no longer require sequences, they can be used in other
ways. Here are some examples.

Finally, transducers can of course be composed, allowing the
building up of logic without requiring them to be used with sequences.