Pipes vs Foldl: where to end one and start the other

Pipes vs Foldl: where to end one and start the other

Hello,

I am rewriting some old code, and I have a design question regarding the "right" way to interface pipes (or generic streams) and folds that may themselves be quite complex. Currently my setup goes something like this (this is pseudocode--haven't checked that it compiles/runs):

- I have one pipe that produces Events, where a simplified event might look like

data Event a b = Event { myA :: a, myBs :: [b] }

- I want to loop over events and store statistics of type r; I do this by implementing Fold(M)s:

foldA :: Fold a r

foldB :: Fold b r

- I combine the statistics using the Applicative or Monoid instances of Fold(M):

foldEvent :: Fold (Event a b) [r]

foldEvent = sequenceA

[ premap myA foldA

, premap myBs $ handles folded foldB

]

- I then connect these stats-gathering folds to my Event Producer, which works quite nicely.

My problem with this setup is one of complexity. In fact my Events are not so simple, and I want to categorize them in arbitrarily complicated ways and then feed them into the r statistics for each category. Currently all the complexity lives in the Fold side of my code, but it seems like Pipes are much better suited to handling non-trivial flows of information. I wonder if it's possible to have a complicated set of Pipes that connect (at the last second) to a bunch of Folds, which I then combine applicatively. There's a lot of Pipes infrastructure that could make the flow easier to understand, and I think the Pipes Monad instance may help clean up a lot of hard-to-read Fold code. In my silly example, Pipes.Prelude.map and Pipes.Prelude.mapFoldable are much easier to reason about (in my mind) than Foldl.premap and "handles folded".

Unfortunately, I can't see how to implement something as simple as my foldEvent using Pipes and still get the correct streaming properties. Is there a solution that I am missing, or should I focus on building up more composable primitives on the Fold side of my program?

Sorry if this is too vague: I don't want to dump a huge codebase on you, but I can provide more examples. Questions and comments welcome!

Re: Pipes vs Foldl: where to end one and start the other

> There's a lot of Pipes infrastructure that could make the flow easier to understand, and I think the Pipes Monad instance may help clean up a lot of hard-to-read Fold code.

Indeed, writing functions that "pull" from a source tends to be easier and more natural that writing "push-style" consumers.

This might not be exactly what you are looking for, but the Streaming.Eversion.Pipes module from my streaming-eversion package has functions that transform operations working on pipes into folds and fold transformers (transducers).

On Tuesday, April 18, 2017 at 6:34:21 PM UTC+2, Chris Pollard wrote:

Hello,

I am rewriting some old code, and I have a design question regarding the "right" way to interface pipes (or generic streams) and folds that may themselves be quite complex. Currently my setup goes something like this (this is pseudocode--haven't checked that it compiles/runs):

- I have one pipe that produces Events, where a simplified event might look like

data Event a b = Event { myA :: a, myBs :: [b] }

- I want to loop over events and store statistics of type r; I do this by implementing Fold(M)s:

foldA :: Fold a r

foldB :: Fold b r

- I combine the statistics using the Applicative or Monoid instances of Fold(M):

foldEvent :: Fold (Event a b) [r]

foldEvent = sequenceA

[ premap myA foldA

, premap myBs $ handles folded foldB

]

- I then connect these stats-gathering folds to my Event Producer, which works quite nicely.

My problem with this setup is one of complexity. In fact my Events are not so simple, and I want to categorize them in arbitrarily complicated ways and then feed them into the r statistics for each category. Currently all the complexity lives in the Fold side of my code, but it seems like Pipes are much better suited to handling non-trivial flows of information. I wonder if it's possible to have a complicated set of Pipes that connect (at the last second) to a bunch of Folds, which I then combine applicatively. There's a lot of Pipes infrastructure that could make the flow easier to understand, and I think the Pipes Monad instance may help clean up a lot of hard-to-read Fold code. In my silly example, Pipes.Prelude.map and Pipes.Prelude.mapFoldable are much easier to reason about (in my mind) than Foldl.premap and "handles folded".

Unfortunately, I can't see how to implement something as simple as my foldEvent using Pipes and still get the correct streaming properties. Is there a solution that I am missing, or should I focus on building up more composable primitives on the Fold side of my program?

Sorry if this is too vague: I don't want to dump a huge codebase on you, but I can provide more examples. Questions and comments welcome!

Re: Pipes vs Foldl: where to end one and start the other

Thanks Daniel. I expected a mail in my inbox, so I just saw this. I'll have a look at your library.

Perhaps a more general question: is there anywhere a guide to what is in Pipes.Core? There are some nice (small) insights on in the haddocks, but it seems like quite a beautiful setup, so I'm wondering if there is a more complete introduction to this module a-la what is in Pipes.Tutorial.

Chris

On Wednesday, May 3, 2017 at 9:16:22 PM UTC+1, Daniel Díaz wrote:

> There's a lot of Pipes infrastructure that could make the flow easier to understand, and I think the Pipes Monad instance may help clean up a lot of hard-to-read Fold code.

Indeed, writing functions that "pull" from a source tends to be easier and more natural that writing "push-style" consumers.

This might not be exactly what you are looking for, but the <a href="http://Streaming.Eversion.Pipes" target="_blank" rel="nofollow" onmousedown="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2FStreaming.Eversion.Pipes\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNHUohx7g-JXBuRhT0iVHAr1ZcZHTA&#39;;return true;" onclick="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2FStreaming.Eversion.Pipes\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNHUohx7g-JXBuRhT0iVHAr1ZcZHTA&#39;;return true;">Streaming.Eversion.Pipes module from my <a href="http://hackage.haskell.org/package/streaming-eversion-0.3.1.1" target="_blank" rel="nofollow" onmousedown="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2Fhackage.haskell.org%2Fpackage%2Fstreaming-eversion-0.3.1.1\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNFNUS39m2M7EikCyvsfR-hsMLzZrA&#39;;return true;" onclick="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2Fhackage.haskell.org%2Fpackage%2Fstreaming-eversion-0.3.1.1\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNFNUS39m2M7EikCyvsfR-hsMLzZrA&#39;;return true;">streaming-eversion package has functions that transform operations working on pipes into folds and fold transformers (transducers).

On Tuesday, April 18, 2017 at 6:34:21 PM UTC+2, Chris Pollard wrote:

Hello,

I am rewriting some old code, and I have a design question regarding the "right" way to interface pipes (or generic streams) and folds that may themselves be quite complex. Currently my setup goes something like this (this is pseudocode--haven't checked that it compiles/runs):

- I have one pipe that produces Events, where a simplified event might look like

data Event a b = Event { myA :: a, myBs :: [b] }

- I want to loop over events and store statistics of type r; I do this by implementing Fold(M)s:

foldA :: Fold a r

foldB :: Fold b r

- I combine the statistics using the Applicative or Monoid instances of Fold(M):

foldEvent :: Fold (Event a b) [r]

foldEvent = sequenceA

[ premap myA foldA

, premap myBs $ handles folded foldB

]

- I then connect these stats-gathering folds to my Event Producer, which works quite nicely.

My problem with this setup is one of complexity. In fact my Events are not so simple, and I want to categorize them in arbitrarily complicated ways and then feed them into the r statistics for each category. Currently all the complexity lives in the Fold side of my code, but it seems like Pipes are much better suited to handling non-trivial flows of information. I wonder if it's possible to have a complicated set of Pipes that connect (at the last second) to a bunch of Folds, which I then combine applicatively. There's a lot of Pipes infrastructure that could make the flow easier to understand, and I think the Pipes Monad instance may help clean up a lot of hard-to-read Fold code. In my silly example, Pipes.Prelude.map and Pipes.Prelude.mapFoldable are much easier to reason about (in my mind) than Foldl.premap and "handles folded".

Unfortunately, I can't see how to implement something as simple as my foldEvent using Pipes and still get the correct streaming properties. Is there a solution that I am missing, or should I focus on building up more composable primitives on the Fold side of my program?

Sorry if this is too vague: I don't want to dump a huge codebase on you, but I can provide more examples. Questions and comments welcome!