The blog post talks about lazy composition only. I haven't mentioned strict composition, as I personally think it's not very useful, as long as you use an appropriate mechanism for resource finalization.

You can also manually append a discard to finite pipes if you want to prevent premature termination of a pipeline.

The reason I like this approach is that it is much easier to reason about and prove that it satisfies the category laws. I was talking to Paolo about it and the final approach will probably still bake this behavior in for performance, but you can still reason about it as above.

Using this guarded pipe composition, you can trivially implement lazy behavior with resource management. The only disadvantage of guarded composition is that the return type is constrained to ().

This approach is indeed easier to reason about, but unfortunately, it still doesn't satisfy the category laws. When you adjust it so that it does (as I did, hopefully, in pipes-extra, it becomes rather involved, and not particularly simpler than modifying the Await primitive directly.

Your id pipe doesn't work because it doesn't terminate when it receives a Nothing. Guarded composition works precisely because the only way a pipe can send a Nothing upstream is by terminating. The identity GPipe must terminate in order to forward the termination upstream.

If you used my identity pipe in your example, both would give the same behavior, namely printing "B". Your second example didn't work because your identity pipe swallowed the Nothing without terminating so it awaited another value and consequently brought down both itself and p. Had it properly terminated upon the first Nothing, then it would have given p a chance to output the "B".

And you are right about yieldMap. It's easier to just replace it with pipe Just <+<.

As far as swallowing return values, you are right that you can't generate folds if the return value is constrained to (). Let me think about it.

I haven't vetted the case statement ordering yet, but you get the idea. The yield and await primitives just use return () as the fallback and you can implement tryYield and tryAwait primitives this way.

In fact, this makes obvious something we both hadn't considered in your initial approach, namely that yields can fail and we were never intercepting them. That would have been a problem for a Producer that had to finalize a resource when its downstream Pipes terminated.

The nice thing about this approach is that you still keep the free monad, which is nice.

You can check out this implementation from the "try" branch of my github repository. I like to use the following pipe for testing finalization:

First of all, the composition of pipes with monadic finalizers, as written, is
not associative. If you have a pipe of the form P >+> A1 >+> A2, the
finalizer of A2 is only going to be called if you associate on the left.

Anyway, I don't think this actually solves the problem. For example, take a
look at the lines conduit
here.
How would you implement this in pipes? If you write something like:

then you lose the last line of the last chunk, which is going to be bound to
leftover, but never yielded, because the pipe immediately terminates when
await fails. Adding only monadic finalizers doesn't help here.

I think it's important not to conflate two distinct notions of finalization.
There is finalization of the pipe's internal state, and finalization of
external resources. The first has a semantic connotation: it is needed so that
the pipe can complete its job before terminating. The second is merely to
prevent leaks and avoid tying up resources unnecessarily.

I believe that those two types of finalizations are orthogonal. There are
techniques to deal with the second type of finalization, like monadic regions
and ResourceT, which can be used together with pipes (see
here
for examples with ResourceT).

Your monadic finalizer approach can be thought of as another solution to the
external finalization problem, but it's incomplete, since it's not
exception-safe. I suppose that using some type-level trickery you could bake
exception-handling logic in the definition of composition, but I don't see the
point of doing that when we already have valid alternatives that work
independently of pipes.

That leads to the second point: handling a failed Yield. As far as an
individual pipe is concerned, yield can never fail. If the downstream pipe
terminates before it receives a certain yielded value, then no more values are
going to be consumed, and there's no reason to keep the pipeline running. The
only reason would be freeing up allocated resources, but again, this can be
handled outside. The asymmetry is there because the pipeline data flow is
unidirectional.

As for the other approach (using a separate monad for finalizer pipes), I think
it's very interesting, but I don't like that we have two completely independent
pipe types, because that hinders composability. Also, again, the composition as
you wrote it is not associative, for similar reasons (you need to handle the
Await + Await case differently).

There is a way to have the best of both worlds (composability and better
statical guarantees): add a type parameter representing whether the pipe is a
finalizer or not, and define Pipe as a GADT:

This is completely untested, but I see no reason why it shouldn't work. It also
makes the distinction between internal and external finalization very apparent:
readFile allocates external resources, but there's nothing wrong in using it
in a finalizer.

In the end, it comes down to the usual debate of whether a little extra static
guarantee is worth forcing an additional parameter in all type declarations. I
don't think you can fairly say that my original variation is not type-safe,
since composition is always well defined, no matter what constructors you use.
It is true that Await has two different semantics according to the context
(it doubles up as Done, basically) and that is not encoded in the types, but
I really think we could live with that.

Yeah, I see your point now. In fact, this is remarkably similar to the dilemma I noted in my documentation tutorial for toList. Any attempt to detect upstream termination fails associativity when you insert a non-terminating Pipe in-between, violating the Category laws. I need to build up a solid intuition for why termination-interception seems to conflict with the Category instance. I'll reply with another comment after I've thought about it a little bit.

Ok, here's another idea I haven't yet fully vetted. So, I basically agree with your overall premise that await has to be modified in order to be able to gracefully intercept shutdowns. What do you think of changing the Await constructor to:

data Pipe a b m r =
...
Await (a -> Pipe a b m r, Pipe a b m r)

The first value is the normal continuation and the second value is the continuation if the upstream pipe terminates before yielding a value. Note that it need not be a finalization routine. It's just a fall-back set of behaviors. Then you can define a few primitives:

Ok, I got it. My thoughts on this have finally crystallized. So I can show you how to improve on your Pipe type and composition instance and use the type system to verify statically that a Pipe will never await twice from a terminated Pipe.

Let's first start from your composition instance. You tag each Pipe with a boolean to indicate whether or not it has awaited once from a terminated pipe. Algebraically, this means that you multiply each Pipe by a boolean:

2 * P

So the first thing I thought was "Why not carry the boolean around with the Pipe in the first place?" (i.e. build it into the Pipe type), since there could in theory be a need to create a Pipe that was already in a finalization state. So in other words, the base PipeF functor becomes:

PipeF = 2 * (A + Y + M)

... where 2 is the boolean tag that tracks whether it has awaited once, A stands for the Await constructors, Y stands for the Yield constructor, and M stands for the M constructor.

So you can represent a product with a boolean as a sum instead:

2 * x = x + x

So we can rewrite our PipeF type as:

PipeF = (A + Y + M) + (A' + Y' + M')

Where the first three constructors correspond to the normal Pipe constructors pre-finalization and the second set of three constructors correspond to the Pipe constructors during finalization.

However, we can statically disallow the Pipe from awaiting a second time by removing the await constructor from the second set:

PipeF = (A + Y + M) + (Y' + M')

And we can also statically enforce that the transition to the finalization states is irreversible at the type-level:

Note that this means that our initial Pipe (pre-finalization) is no longer a free monad. This is the insight I was missing because I kept trying to figure out how it was possible to reconcile a free monad with intercepting shutdown. This was also the objection I initially had with your composition instance: It doesn't enforce the irreversibility of the transition at the type level.

Also note that the above definitions are incompatible with your original Await constructor. I distinguish the two continuations at the type level in order to enforce irreversibility so you can't unify them with a Maybe argument any longer.

This leads to the type-safe and (in my opinion) much more elegant composition that represents the nub of finalization:

Now that we typefully distinguish between pre-finalization and finalization we can now define the correct await function:

await = Await (Pure, Done)

And our identity pipe is still the same as ever:

id = forever $ await >>= yield

Since we introduced a new type for the finalization phase, we have to introduce two new primitive for the constructors during finalization:

yield' x = Yield' (x, Pure ())
done = Done -- just so we don't have to expose the constructor, and also because I like lower-case
-- Pure' is covered by the Monad instance for Pipe'
-- M' is covered by the MonadTrans instances for Pipe'

Also, would you be interested in coauthoring a paper on this? I think now that Pipes has type-safe, yet lazy, resource management it is publication-worthy (and ready for production code, too). I must admit I'm still a novice at computer science (since I'm a graduate student in bioinformatics) so I'm not entirely sure if this qualifies as publishable or not, but I'm pretty sure it does.

They are indeed isomorphic constructors and your conversion functions define the isomorphism. I like your constructor much better since it's considerably easier to implement all the class instances for it and it's more compact.

Syntactically, I prefer my version of the tryAwait function since it abstracts away the Maybe, although we could provide both functions. Also, I don't really like the name tryAwait and I'll try to think of a sexier name for both your version and my version.

I still think there is something messy about your composition. I'm trying to play around with different await and composition definitions to get something more elegant.