You can define your own lazy components that have nothing to do with files

pipes never uses unsafePerformIO and never violates referential
transparency.

You don't need strictness hacks to ensure the proper ordering of effects

You can interleave effects in downstream stages, too

However, this library can offer even more than that!

Bidirectionality

So far we've only defined proxies that send information downstream in the
direction of the (>->) arrow. However, we don't need to limit ourselves
to unidirectional communication and we can enhance these proxies with the
ability to send information upstream with each request that determines
how upstream stages respond.

For example, Clients generalize Consumers because they can supply an
argument other than () with each request. The following Client
sends three requests upstream, each of which provides an Intargument
and expects a Boolresult:

This bidirectional flow of information separates pipes from other
streaming libraries which are unable to model Clients, Servers, or
Proxys. Using pipes you can define interfaces to RPC interfaces, REST
architectures, message buses, chat clients, web servers, network protocols
... you name it!

Type Synonyms

You might wonder why (>->) accepts Producers, Consumers, Pipes,
Clients, Servers, and Proxys. It turns out that these type-check
because they are all type synonyms that expand to the following central
type:

(Proxy p) => p a' a b' b m r

Like the name suggests, a Proxy exposes two interfaces: an upstream
interface and a downstream interface. Each interface can both send and
receive values:

respond replies with a result of type b, and then binds the next
argument of type b'. Whenever you respond, you block until downstream
requests a new value.

Wait, if respond always binds the next argument, where does the first
argument come from? Well, it turns out that every Proxy receives this
initial argument as an ordinary parameter, as if they all began blocked on
a respond statement.

We can see this if we take all the previous proxies we defined and fully
expand every type synonym. The initial argument of each Proxy matches
the type parameter corresponding to the return value of respond:

... then everything would still work and produce identical behavior, except
the compiler would now infer the symmetric types with all interfaces
reversed. We can therefore conclude the obvious: pull-based systems are
symmetric to push-based systems.

Since these two composition operators are perfectly symmetric, I arbitrarily
standardize on using (>->) and I provide all standard library proxies
blocked on respond so that they work with (>->). This gives behavior
more familiar to Haskell programmers that work with lazy pull-based
functions. I only include the (>~>) composition operator for theoretical
completeness.

Composition

When we compose (p1 >-> p2), composition ensures that p1's downstream
interface matches p2's upstream interface. This follows from the type of
(>->):

This means that proxies form a true Category where (>->) is composition
and idT is the identity. The associativity law and the two
identity laws are just the Category laws. The objects of the category are
the Proxy interfaces.

These are not the only instances of the Proxy type class! This library
also provides several "proxy transformers", which are like monad
transformers except that they also correctly lift the Proxy type class:

All of the Proxy code we wrote so far also works seamlessly with all of
these proxy transformers. The Proxy class abstracts over the
implementation details and extensions so that you can reuse the same library
code for any feature set.

This polymorphism comes at a price: you must embed your Proxy code in at
least one proxy transformer if you want clean type class constraints. If
you don't use extensions then you embed your code in the identity proxy
transformer: IdentityP. This is why all the examples use runIdentityP
or runIdentityK to embed their code in IdentityP. Control.Proxy.Class
provides a longer discussion on this subject.

Without this IdentityP embedding, the compiler infers uglier constraints,
which are also significantly less polymorphic. We can show this by
removing the runIdentityP call from promptInt and see what type the
compiler infers:

All Proxy instances are already monads and monad transformers, but the
compiler cannot infer that without the IdentityP embedding. When we embed
promptInt in IdentityP, the compiler collapses the Monad and
MonadTrans constraints into the Proxy constraint.

Fortunately, you do not pay any performance price for this IdentityP
embedding or the type class polymorphism. Your polymorphic code will still
run very rapidly, as fast as if you had specialized it to a concrete
Proxy instance without the IdentityP embedding. I've taken great care
to ensure that all optimizations and rewrite rules always see through these
abstractions without any assistance on your part.

Interleaving Effects

When you compose two proxies, you interleave their effects in the base
monad. The following two proxies demonstrate this interleaving of effects:

Several of the utilities in Control.Proxy.Prelude.Base use these
equational laws to rigorously prove things about their behavior. For
example, consider the mapD proxy, which applies a function f to all
values flowing downstream:

... which is what we expect. We can fuse two consecutive mapDs into one
by composing their functions, and mapping id does nothing at all, just
like the identity proxy: idT.

In fact, these are just the functor laws in disguise, where mapD defines a
functor between the category of Haskell function composition and the
category of Proxy composition. Control.Proxy.Prelude.Base is full of
utilities like this that are simultaneously practical and theoretically
elegant.

Mixing Base Monads

Composition can't interleave two proxies if their base monads do not
match. For instance, I might try to modify promptInt to use
EitherT String to report the error instead of using exceptions:

The type error says that promptInt2 uses (EitherT String IO) for its
base monad, but printer uses IO for its base monad, so composition can't
interleave their effects.

You can easily fix this using the hoist function from the mmorph
package, which transforms the base monad of any monad transformer that
implements MFunctor. Since all proxies implement MFunctor you can use
hoist from MFunctor to lift one proxy's base monad to match another
proxy's base monad, like so:

printD has a more general type than our original printer because it
forwards all values further downstream after printing them. This means
that you could use it as an intermediate stage as well. However, printD
still type-checks as the most downstream stage, too, since runProxy just
discards any unused outbound values.

These utilities do not clash with the Prelude namespace or common libraries
because they all end with a capital letter suffix that indicates their
directionality:

Fortunately, we don't have to give up our previous useful diagnostics.
We can use execU, which executes an action each time values flow upstream
through it, and execD, which executes an action each time values flow
downstream through it:

You assemble most proxies simply by composing them in one or both of these
two categories.

ListT

Proxies generalize lists by allowing you to interleave effects between list
elements, but you might be surprised to learn that they also generalize the
list monad, too! Control.Proxy.ListT provides the machinery necessary to
convert back and forth between proxies and a ListT-like monad transformer
that binds proxy outputs.

For example, let's say that we want to select elements from two separate
lists, except interleaving side effects, too:

Proxies actually form two symmetric ListT-like monad transformers: one
binds elements output from the proxy's downstream interface and one binds
elements output from the proxy's upstream interface. To distinguish them,
I call the downstream one RespondT and the upstream one RequestT.

You don't need to use these operators directly, since you can instead just
wrap your proxies in the RespondT or RequestT monad transformers and
they will translate the binds in those monads to the above operators under
the hood.

If those are the bind operators, though, then where are the corresponding
return equivalents? Why, they are just respond and request!

RequestT and RespondT are correct by construction, meaning that they
always satisfy the monad and monad transformer without exception, unlike
ListT from transformers. In other words, they behave like two
symmetric implementations of "ListT done right".

Folds

You can fold a stream of values in two ways, both of which use the base
monad:

WriterT is more elegant in principle but leaks space for a large number of
values to fold. StateT does not leak space if you keep the accumulator
strict, but is less elegant and doesn't guarantee write-only behavior. To
remedy this, I am currently working on a stricter WriterT implementation
that does not leak space to add to the transformers package.

These WriterT versions demonstrate how the elegant approach should work in
principle and they should be okay for folding a medium number of values
until I release the fixed WriterT. If space leaks cause problems, you can
temporarily rewrite the WriterT folds using the following two strict
StateT folds:

I designed certain special folds to terminate the Session early if they
can compute their result prematurely, in order to draw as little input as
possible. These folds end with an underscore, such as headD_, which
terminates the stream once it receives an input:

Use the versions that don't prematurely terminate if you are running
multiple folds or if you want to continue to use the rest of the input when
the fold is done. Use the versions that do prematurely terminate if
collecting that single fold is the entire purpose of the session.

Resource Management

This core library provides utilities for lazily streaming from resources,
but does not provide utilities for lazily managing resource allocation and
deallocation. To frame the problem, let's assume that we try to be clever
and write a streaming utility that lazily opens a file only in response to
a request, such as the following Producer:

Notice that this does not close the file, because once takeB_ 1 terminates
it terminates the entire Session and readFileS does not get a chance to
finalize the file.

The pipes-safe library solves this problem by providing resource
management abstractions built on top of pipes and offers several other
nice features:

It is completely exception safe, even against asynchronous exceptions

It is backwards compatible with "unmanaged" ordinary proxies

Backwards compatibility means that you don't need to buy in to the
pipes-safe way of doing things. This matters because another common
approach is to just open all resources before running the session and close
them all afterward. For example,, if I wanted to emulate the Unix cp
command, streaming one line at a time, I might write:

The disadvantage is that this does not lazily allocate resources, nor does
this promptly deallocate them. Also, there is no way to recover from
exceptions and resume the Session. On the other hand, pipes-safe lets
you do all of these.

Fortunately, you can choose whichever approach you prefer and rest assured
that the two approaches safely interoperate. Control.Proxy.Safe.Tutorial
from the pipes-safe package provides a separate tutorial on how to:

extend pipes with resource management,

handle exceptions natively within proxies, and

interoperate with unmanaged code.

Extensions

This library provides several extensions that add features on top of the
base Proxy API. These extensions behave like monad transformers, except
that they also lift the Proxy class through the extension so that the
extended proxy can still request, respond, and compose with other
proxies:

Notice how we can directly compose printer with promptInt.
This works because printer's base proxy type is completely polymorphic
over the Proxy type class and doesn't use any features specific to any
proxy transformers:

They each share the same initial state, but they isolate their own side
effects completely from each other.

Branching, zips, and merges

So far we've only considered linear chains of proxies, but pipes allows
you to branch these chains and generate more sophisticated topologies. The
trick is to simply nest the Proxy monad transformer within itself.

For example, if I want to zip two inputs, I can just define the following
triply nested proxy:

We just forked a (StateP p1) proxy and read out the result in both a
generic p2 proxy and an (EitherP p3) proxy. That's pretty crazy, but it
gives you a sense of how versatile and robust proxies can be.

You can implement arbitrary branching topologies using this trick. However,
I want to mention a few caveats:

The intermediate partially applied type signatures will be ugly as sin.
I warned you.

You cannot implement cyclic topologies (and cyclic topologies do not make
sense for proxies anyway)

You cannot use this trick to implement a polymorphic zip function of the
following form:

Partial application requires selecting a Proxy instance, which is why you
cannot define zip'. You can define a zip' specialized to a concrete
Proxy instance, but I don't really recommend doing that since you should
always strive to write polymorphic proxies to avoid locking your user into
a particular feature set.

With those caveats out of the way, this approach affords many indispensable
features that other approaches do not allow:

It is completely polymorphic over the Proxy class and uses no
implementation-specific details

Proxy Transformers

There is one last scenario that you will eventually encounter: mixing
proxies that have incompatible proxy transformer stacks. You solve this the
exact same way you mix different monad transformer stacks, except that
instead of using lift and hoist you use liftP and hoistP.

Like monad transformers, proxy transformers lift a base Monad instance
to an extended Monad instance and liftP exactly mirrors the lift
function from MonadTrans. liftP takes some base proxy, p, that
implements Monad and "lift"s it to an extended proxy, (t p), that also
implements Monad.

This means you can seamlessly mix effects from different proxy transformer
layers just by using liftP to access inner layers:

However, proxy transformers do one extra thing above and beyond ordinary
monad transformers. Proxy transformers lift the Proxy type class, meaning
that if the base type implements Proxy, so does the extended type.

This means that we need a set of laws that guarantee that the proxy
transformer lifts the Proxy instance correctly. I call these laws the
"proxy morphism laws":

However, I don't expect everybody to immediately understand how so few
primitives can implement such a wide variety of features. This tutorial
gives a taste of how many interesting ways you can combine these few
abstractions, but these examples barely scratch the surface, despite this
tutorial's length. So if you don't know how to implement something using
pipes, just ask me and I will be happy to help.

Appendix

I've provided the full code for the above examples here so you can easily
play with the examples yourself: