The Essence of the Iterator Pattern

In this post I'll try to present some ideas of that paper and show how to implement the solution described by the authors using Scalaz-like code. A minimum previous exposure to functional programming, functors and monads will definitely help!

What's in a for loop?

That was really what hooked me. What do you mean "what's in a for loop"?. Is there anything magic in that construct I've been using for years?

The introduction of EIP shows an example of a for loop to iterate on elements (not the C-like for used with an index). I'm transmogrifying it here to Scala but the idea remains the same:

We start from a "container" of fruits: Basket. It could actually be anything, a List, a Tree, a Map... Then the for loop actually does 3 things:

it returns a container having the same "shape"; juices is still a Basket

it accumulates some kind of measure. Here, the number of fruits in the count variable

it maps the elements to other elements: pressing the fruits to get some juice

And this for loop is actually not the most complex:

the count variable could influence the mapping of elements: juices.add(fruit.press(harder=count))

we could have several variables depending on each other: cumulative = cumulative + count

the mapping could also influence a "measure" variable: liquid = liquid + fruit.press.quantity

The purpose of EIP is to show that the "essence" of what happens in the for loop above can be abstracted by an Applicative Traversal. And the authors go on showing that given this Applicative abstraction, we get an incredible modularity for programming.

The Applicative typeclass

How can an Applicative traversal be better than a for loop, and what does that even mean?? EIP has a lot of sentences and expressions which can be hard to grasp if you don't have a strong functional programming / Haskell background. Let's try to dissect that slowly and start with the formal definitions anyway.

What is a Functor?

The first thing we need to talk about is the Functor typeclass:

trait Functor[F[_]] { def fmap[A, B](f: A => B): F[A] => F[B] }

One way of interpreting a Functor is to describe it as a computation of values of type A. For example List[A] is a computation returning several values of type A (non-deterministic computation), Option[A] is for computations that you may or may not have, Future[A] is a computation of a value of type A that you will get later, and so on. Another way of picturing it is as some kind of "container" for values of type A.

Saying that those computations are Functors is essentially showing that we can have a very useful way of combining them with regular functions. We can apply a function to the value that is being computed. Given a value F[A] and a function f, we can apply that function to the value with fmap. For example, fmap is a simple map for a List or an Option.

Pointed Functor

By the way, how do you even create a value of type F[A]? One way to do that is to say that F[_] is Pointed:

trait Pointed[F[_]] { def point[A](a: => A): F[A] }

That is, there is a point function taking a value of type A and returning a F[A]. For example, a regular List is Pointed just by using the constructor for Lists:

The PointedFunctor trait is merely the aggregation of a Pointed and a Functor.

What about Applicative then? We're getting to it, the last missing piece is Applic.

Applic

Applic is another way to combine a "container" with a function.

Instead of using fmap to apply the function to the computed value we suppose that the function is itself a computed value inside the container F (F[A => B]) and we provide a method applic to apply that function to a value F[A]:

trait Applic[F[_]] { def applic[A, B](f: F[A => B]): F[A] => F[B] }

Let's take an example. Say I have a way to compute the price of a Fruit when the market is open:

def pricer(market: Market): Option[Fruit => Double]

If the market is closed, pricer returns None, because we don't know what are the prices. Otherwise it returns a pricing function. Now if I have a grow function possibly returning a Fruit:

def grow: Option[Fruit]

Then, using the Applic instance, you can price the Fruit:

val price: Option[Double] = applic(pricer(market)).apply(grow)

The price will necessarily be an Option because you may not have a pricer nor a Fruit to price. And a bit of renaming and pimping reveals why we're using the term "Applicative":

val pricingFunction = pricer(market) val fruit = grow

val price: Option[Double] = pricingFunction ⊛ fruit

In a way we're just doing a normal function application, but we're just doing it inside the Applicative container. Now we have all the pieces to build the Applicative functor that EIP is talking about.

Applicative Functor

An Applicative Functor is the aggregation of an Applic and a PointedFunctor:

Informally the traverse method applies the function f to each node and "reconstructs" the tree by using the apply method (<*>) of the Applicative functor.

That's certainly still some Ancient Chinese to you (as it was for me) so we'd better see the traverse method at work now. But we need to take another detour :-)

Applicative Monoid

One simple thing we might want to do, when iterating on a BinaryTree, is to get the content of that tree in a List. To do that, we're going to use the 3rd way to use List as an Applicative, as mentioned earlier. It turns out indeed that any Monoid (what is it?) gives rise to an Applicative instance but in a way that's a bit surprising.

/** Const is a container for values of type M, with a "phantom" type A */ case class Const[M, +A](value: M)

In the code above, Const is the Applicative instance for a given Monoid. Const contains values of type T where T is a Monoid and we progressively establish what are the properties that Const must satisfy to be Applicative.

it must first be Pointed. Informally, the point method puts the neutral element of the Monoid in a Const instance

then it must be a Functor. Here the fmap function doesn't do anything but changing the type of Const from Const[M, A] to Const[M, B]

finally it must be an Applic where the apply method of Applic uses the append method of the Monoid to "add" 2 values and return the result in a Const instance.

There is unfortunately a lot of typing vodoo thing here:

the type declaration for Const is Const[A, +B]. It has a type parameter B which is actually not represented by a value in the Const class! It is a phantom type. But it is actually indispensable to match the type declarations of the typeclasses

the type F that is supposed to be Applicative is... ({type l[A] = Const[T, A]})#l. Ouch, this deserves some explanation!

What we want is not so hard. The type Const[A, B] has 2 type parameters. We need a way to fixA to be T and get the resulting type which will have only one type parameter. The expression above is the most concise way to get this desired type:

{ type l = SomeType } is an anonymous type with a type member called l. We can access that type l in Scala by using #: { type l = SomeType }#l

Then, in { type l[A] = SomeType[T, A] }#l, l is a higher-kinded type, having a type variable A (actually SomeType[T, A] where T is fixed)

That was a really long detour for a mere for loop, isn't it? Now... profit!

Contents of a BinaryTree...

We're going to use the Traversable instance for the BinaryTree and the List Monoid Applicative to get the contents of a BinaryTree:

import Applicative._

val f = (i: Int) => List(i) val tree = Bin(Leaf(1), Leaf(2))

(tree.traverse[...](f)).value must_== List(1, 2)

Simple, for each element of the tree, we put it in a List then we let the List Monoid do its magic and aggregate all the results as we traverse the tree. The only difficulty here is the limits of Scala type inference. The ... stands for type annotations that the compiler requires:

tree.traverse[Int, ({type l[A]=Const[List[Int], A]})#l](f)

Not pretty :-(

Update: As pointed out by Ittay Dror in the comments, List[Int] is not an applicative by itself and we need to put this list into a Const value to make it usable by the traverse function.

This is actually done by an implicit conversion method, liftConst, provided by the Applicative object:

Profit time

Not everything is lost! We can encapsulate a bit the complexity in this case. We can extract part of the code above and create a contents method which will work on any of Traversable instance (assume I'm pimping the following examples so that I can write tree.method instead of method(tree)):

The reduce function can traverse any Traversable structure with a function mapping each element to a Monoid element. We've used it to get the contents of the tree but can as easily get the number of elements:

def count[A]: T[A] => Int = reduce((a: A) => 1)

tree.count must_== 2

Can it get simpler than this :-)? Actually in that case it can! Since we don't need (a: A) at all we can use reduceConst:

def reduceConst[A, M : Monoid](m: M): T[A] => M = reduce((a: A) => m)

def count[A]: T[A] => Int = reduceConst(1)

It's like a Scala standard reduce on steroids because instead you don't need to provide a binary operation, you just need a Monoid instance.

.... and shape of a BinaryTree

We've addressed the question of doing some kind of accumulation based on the elements in the tree, now we're going to "map" them.

Monads are Applicatives too!

The following map method can be derived from the traverse method (note that no type annotations are necessary in that case, yes!):

We can even use it to get the "shape" of our container and discard all the elements:

tree.shape must_== Bin(Leaf(()), Leaf(()))

The shape method just maps each element to ().

Decompose / Compose

I recap. We implemented a very generic way to iterate over a structure, any kind of structure (as long as it's Traversable), containing elements, any kind of element, with a function which does an "application", any kind of application. Among the possible "applications", we've seen 2 examples: collecting and mapping which are the essential operations that we usually do in a for loop.

Specifically we were able to get the contents of a tree and its shape. Is there a way to compose those 2 operations into a decompose operation that would get both the content and the shape at once? Our first attempt might be:

def decompose[A] = (t: T[A]) => (shape(t), contents(t))

tree.decompose must_== (Bin(Leaf(()), Leaf(())), List(1, 2))

This works but it is pretty naive because this requires 2 traversals of the tree. Is that possible to do just one?

Applicative products

This is indeed possible by noticing the following: the product of 2 Applicatives is still an Applicative.

Then no implicit val productApplicative is necessary and the Applicative imports will be all we need.

Collection and dispersal

There is another way to do things "in parallel" while traversing the structure. The collect method that we're going to build will do 2 things:

it will accumulate some kind of state, based on the elements that we meet

it will map each element to another kind of element

So, as we're iterating, we can do a regular mapping while computing some kind of measure. But before that, we need to take a little detour (again?? Yes, again) with the State monad.

The State monad

The State Monad is defined by:

trait State[S, +A] { def apply(s: S): (S, A) }

It is basically:

an object keeping some previous "state", of type S

a method to extract a meaningful value from this "state", of type A

this method computes a new "state", of type S

For example, a simple counter for the number of elements in a List[Int] can be implemented by:

val count = state((n: Int) => (n+1, ()))

It takes the previous "count" number n and returns the new state n+1 and the extracted value (() here, because we don't need to extract anything special).

The State type above is a Monad. I encourage you to read "Learn You a Haskell" to get a better understanding on the subject. I will just show here that the flatMap (or bind) method of the Monad typeclass is central in putting that State to work:

The count function takes the latest computed string and returns a State where we increment the current "state" by 1 and we have a new String as the result, where the current count is appended. So when we start with the string "a-" and we flatMapcount 2 times, we get (3, "a-012") where 3 is the number of times we've applied the n+1 function and "a-012" the result of appending to the current string.

By the way, why do we need to apply(0)?

When we do all the flatMaps, we actually store "stateful computations". And they are executed only once we provide the initial state: 0!

Collecting elements

Let's now define a collect operation on Traversable which will help us to count:

I don't know about you, but I find this a bit magical. With the Applicative and Traversable abstractions, we can assemble our program based on 2 independent functions possibly developed and tested elsewhere.

Dispersing elements

The next utility function proposed by EIP is the disperse function. Its signature is:

It is using the very capabilities of the applicative functor, the point method and the <*> application.

An overview of traversals

We've seen in the 2 examples above that we get different, specialized, versions of the traverse function by constraining how mapping and Applicative effects occur. Here's a tentative table for classifying other specialized versions of the traverse function:

function

map element

create state

mapped depend on state

state depend on element

collect

X

X

X

disperse

X

X

X

measure

X

X

traverse

X

X

X

X

reduce

X

X

reduceConst

X

map

X

The only function we haven't shown before is measure. It is mapping and accumulating state but this accumulation does not depend on the current element. Here's an example:

Other than not looking very useful, the code above is also lying! It is not possible to have a measure function accepting a State monad without having to provide the usual ugly type annotations. So the actual example is:

where measureState is a specialization of the measure method to States. I think that one take-away of this post is that it might be beneficial to specialize a few generic functions in Scala , like traverse, collect... for Const and State in order to avoid type annotations.

In the body of that for loop, you have statements with dependency on each other. In an Applicative traversal, this translates to the Sequential composition of Applicatives. From 2 Applicatives, we can create a third one which is their Sequential composition. More precisely, this means that if F1[_] and F2[_] are Applicatives then F1[F2[_]] is an Applicative as well. You want the demonstration? Ok, go.

liftA2 allows to lift a regular function of 2 arguments to a function working on the Applicative arguments. This is using the fact that an ApplicFunctor is a Functor so we can apply function: A => B => C to the "a in the box", to get a F[B => C] "in the box". And then, an ApplicFunctor is an Applic, so we can "apply" F[B] to get a F[C]

Armed with this function, we can write the applic method for F1[F2[_]]:

It's not so easy to get an intuition for what the code above is doing except that saying that we're using previous definitions to allow a F1[F2[A => B]] to be applied to F1[F2[A]].

In mere mortal terms, this means that if we do an Applicative computation inside a loop and if we reuse that computation in another Applicative computation, we still get an Applicative computation. The EIP illustration of this principle is a crazy function, the assemble function.

The assemble function

The assemble function takes the shape of a Traversable and a list of elements. If there are enough elements it returns Some[Traversable] filled with all the elements (+ the reminder), otherwise it returns None (and an empty list). Let's see it in action:

The takeHead function is a State instance where each state application removes the first element of the list of elements if possible, and returns it in an Option.This is why the result of the assemble function, once we apply it to a list of elements, is of type (List[Int], Option[BinaryTree[Int]]).

A recursive implementation

Just for the fun of comparison, I'm going to write a recursive version doing the same thing:

A classical for-loop implementation

By the way, what would be the realfor loop version of that functionality? That one is not so easy to come up with because AFAIK there's no easy way to iterate on a BinaryTree to get a similar BinaryTree with just a for loop! So, for the sake of the argument, we're going to do something similar with just a List structure:

The Applicative composition is indeed very powerful, but we're going to see that there are other ways to compose functions and use them with Traversables.

Monadic composition

This paragraph is exploring the fine relationships between applicative composition and monadic composition when doing traversals. We've seen that Applicative instances can be composed and that Monads can be Applicative. But Monads can also be composed using the so-called Kleisli composition. If we have:

val f: B => M[C] val g: A => M[B]

Then

val h: A => M[C] = f ∎ g // is also a function from a value to a Monad

If we have 2 "monadic" functions f and g, we can then compose them, in the Kleisli sense, and use the composed version for a traversal. Indeed we can, but does this traversal have "nice properties"? Specifically, do we have:

traverse(f ∎ g) == traverse(f) ∎ traverse(g)

The answer is... it depends.

Monad commutativity

EIP shows that, if the Monad is commutative, then this will always be true. What is a commutativeMonad you ask?

However it is obvious that plus1 and plus2 are commutative. What does that mean when we do a traversal?

If we traverse a simple List of elements using monadic composition we get:

List(1, 2, 3).traverse(times2 ∎ plus1) === 22

List(1, 2, 3).traverse(times2) ∎ List(1, 2, 3).traverse(plus1) === 32

We get different results. However, when f and g commute we get the same result:

List(1, 2, 3).traverse(plus2 ∎ plus1) === 10

List(1, 2, 3).traverse(plus2) ∎ List(1, 2, 3).traverse(plus1) === 10

Applicative composition vs Monadic composition

Another question we can ask ourselves is: if we consider the monadic functions as applicative functions (because each Monad is Applicative), do we get the nice "distribution" property we're after? The answer is yes, even when the functions are not commutative:

List(1, 2, 3).traverse(times2 ⊡ plus1) === 4

List(1, 2, 3).traverse(times2) ⊡ List(1, 2, 3).traverse(plus1) === 4

Well... more or less. The real situation is a bit more complex. List(1, 2, 3).traverse(times2 ⊡ plus1) returns a State[Int, State[Int, List[Int]]] while the second expression returns a State[Int, List[State[Int, Int]] so what I'm hiding here is some more manipulations to be able to query the final result with some kind of join.

Conclusion

You wouldn't believe it but I've only shown here half of the ideas presented in EIP!

To finish off this post here's 3 take-away points that I've learned while writing it:

functional programming is also about mastering some of these higher-level control structures like Applicative. Once you master them, your toolbox expands considerably in power (just consider the assemble example)

Scalaz is an incredible library but somewhat obscure to the beginner. For this post I've rewritten all the typeclasses I needed to have, and lots of examples (using specs2 of course). That gave me a much better understanding of the Scalaz functionality. You may consider doing the same to learn Scalaz (my code is available on github)

Scala is lacking behind Haskell in terms of type inference and it's a real pain for higher-order, generic programming. This can be sometimes encapsulated away by specializing generic functions to very common types (like traverseState instead of traverse). Again, please upvote SI-2712!

One minor problem though. (I hope this inspires you to another post on type class laws, Scalacheck, and laziness!).

Your second Applicative definition for List (zipping the elements and functions) doesn't satisfy the Applicative Law.

See why here: https://gist.github.com/1047343

The fix is to define pure (aka point) differently in this case. In the example, we needed the pure function g replicated three times in a list, to zip with all elements of `x`. But how can we generalize this, so we don't need prior knowledge of the length of `x`? Well, just pick a very large number: ∞! This is allowed with Haskell Lists, which are closer to Scala Streams.

I just checked Scalaz; and we actually make the same mistake. Our `newtype` for the zippy Applicative Functor, ZipStream, is correctly based on Stream. But the definition of Pure is incorrect.

Here are the Haskell definitions of ZipList and its type class instances.

Hi Jason, I indeed had bad vibes when writing this definition and forgot to check that.

A follow-up might be a good idea some day. I would like in particular to show why thinking about laws and properties and laws is a good thing, for very practical purposes and not esoteric mathematical leisure. For example "fusion" laws are important for optimization reasons.

Any chance you could provide some further detail or reference material for the "({type l[A]=Product[F1, F2, A]})#l" type statements? It's clear what they do, but going on to create similar constructs still feels like a black art.

I was hoping that my description was clear, but let me try again from another angle, which is how to create one when you need it.

Let's say you have a higher-kinded type: T[A, B, C] and a function which only accepts a higher-kinded type M[_].

Given your type T[A, B, C] you know that if you fix A to be Int and B to be String (for example) then T[Int, String, _] is the M[_] you're looking for. For example, for a State Monad, S[A, B], you know it is a Monad when you fix A to something, say Int, and just use B as the type parameter:

({type l[B]=State[Int,B]}#l)

Then, this type 'l' can have a corresponding Monad instance.

This operation of fixing some type parameters to some known types and letting one other type vary is a Partial type application. Unfortunately, there is not direct way to do that with Scala. The indirect way is to create a type alias and refer to that type.

So if we take the example of having T[A, B, C] and fixing A to Int, B to String, that gives:

({type l[C]=T[Int, String, C]})#l

Where 'l' is an alias name for the higher-kind type T[Int, String, C] with just one type parameter, C.

To be honest, I find too that there's a lot of syntax that needs to be remembered in this formula. The way I did it was by really understanding each part:

- '{}' to create an anonymous class

- 'type' to declare a type member inside an object. It is also a type alias, because it is '=' to another type

- '#' to access the type member of a class

I hope that I'm not too off-base with the exact terms that you can find in the Scala specification and that you'll be able to create such a type by yourself next time. Actually you can grab the code on github and try to reimplement by yourself the Traversable.reduce method in terms of the Traversable.traverse method for example.

It's possible that there is an error on the List implementation of the Applicative Functor? I'm referring to the implementation of 'applic' that zip the List[A] to the List[A=>B]. Shouldn't be p._2 apply p._1?

I think it would be also good to mention liftConst since this is what causes f: Int => List[Int] to be used as a Const in the first traverse example and is crucial since it puts the value of List(i) into the const (the methods that make Const into an applicative use either Monoid.z or Monoid.append and it is not clear where other values come from)

@Benoit, yes it's a call-by-name parameter and it's particularly important if the Functor is a Future for example. Because in that case you want the value to be evaluated in another thread and not right away.

I understand that Scalaz is a bit of a pre-requisite here, but this article would be much easier to follow if you had presented Monoid like you did the other basic concepts. It seems to be the only relevant concept whose interface/definition was not presented.

Also... could you please redefine Const as "case class Const[M, +A](value: M)"? I have a terrible time dissociating the "A" used in the original definition from the "A" type parameters used on the following definitions.

There is no type error on the collect operation, it indeed has only 2 type parameters, one for the applicative (an Int counter using State) and one for the result of "map" operation. The Int type parameter you are tempted to add is in a sense already there because it is in the type of the binary tree that owns the collect operation.

Not that I trust myself that much btw, that's just what the compiler tells me :-)