Saturday, May 4, 2013

Program imperatively using Haskell lenses

Haskell gets a lot of flack because it has no built-in support for state and mutation. Consequently, if we want to bake a stateful apple pie in Haskell we must first create a whole universe of stateful operations. However, this principled approach has paid off and now Haskell programmers enjoy more elegant, concise, and powerful imperative code than you can find even in self-described imperative languages.

Lenses

Your ticket to elegant code is the lens library. You define your data types as usual, but you prefix each field with an underscore. For example, I can define a Game:

We prefix these fields with an underscore because we will not be using them directly. Instead, we will use them to build lenses, which are much more pleasant to work with.

We can build these lenses in two ways. Our first option is to define lenses manually using the lens convenience function from Control.Lens. For example, we can define a score lens to replace the _score field accessor:

A Lens is like a map which you use to navigate complex data types. We use the above score lens to navigate from our Game type to its _score field.

The type reflects where we begin and end: Lens' Game Int means we must begin on a value of type Game and end on a value of type Int (the score, in this case). Similarly, our other lenses will clearly indicate their starting and ending points in their types:

strike prints an evocative sound to the console, then decrements the boss's health by 10 hit points.

strike's type indicates that it operates within the StateT Game IO monad. You can think of this as a DSL where we layer our pure game state (i.e. StateT Game) on top of side effects (i.e. IO) so that we can both mutate our game and also print cute battle effects to the console. All you have to remember is that any time we need side effects, we will use lift to invoke them.

We'll test out strike in ghci. In order to run strike, we must supply it with an initialState:

execStateT takes our stateful code and an initial state, and then runs that code to produce a new state. ghci automatically shows the return value as a convenience so we can inspect the newly returned state. The output is a bit of a mess, but if you strain your eyes you can see that the boss now only has 90 health.

We can view this more easily by storing the new state in a variable:

>>> newState
... and then we can query newState for the part we actually care about:

>>> newState^.boss.health
90

Composition

This syntax very strongly resembles imperative and object-oriented programming:

boss.health -= 10

What is going on here? Haskell is decidely not a multi-paradigm language, yet we have what appears to be multi-paradigm code.

Amazingly, nothing on that line is a built-in language feature!

boss and health are just the lenses we defined above

(-=) is an infix function

(.) is function composition from the Haskell Prelude!

Wait, (.) is function composition? Really?

This is where the lens magic comes in. Lenses are actually ordinary functions, and our "multi-paradigm" code is actually functions all the way down!

In fact, Lens' a b is actually a type synonym for a certain type of higher-order function:

type Lens' a b =
forall f . (Functor f) => (b -> f b) -> (a -> f a)

You don't need to understand the details of that. Just remember that Lens' a b is a higher-order function that accepts a function of type (b -> f b) as an argument, and returns a new function of type (a -> f a). The Functor part is the theoretically-inspired "magic".

Armed with that knowledge, let's make sure the types check out by expanding out the Lens' type synonyms for boss and health

traversed lets us "dig in" to the values in a list so that we can manipulate them as a single unit instead of manually looping over the list. However, this time the type is a Traversal' instead of a Lens'.

>>> newState >> newState^.partyHP
<interactive>:3:11:
No instance for (Data.Monoid.Monoid Int)
arising from a use of `partyHP'
Possible fix:
add an instance declaration for (Data.Monoid.Monoid Int)
In the second argument of `(^.)', namely `partyHP'
In the expression: newState ^. partyHP
In an equation for `it': it = newState ^. partyHP

Oops! This is a type error because there is no single health to get! This is why a Traversal' is weaker than a Lens': traversals may point to multiple values, so they do not support a well-defined way to get just one value. The type system saved us from a potential bug!

Instead, we must specify that we actually want a list of values using the toListOf function:

toListOf :: Traversal' a b -> a -> [b]

This gives the desired result:

>>> toListOf partyHP newState
[7,12,5]

... and there's an infix operator equivalent to toListOf: (^..):

>>> initialState^..partyHP
[10,15,8]
>>> newState^..partyHP
[7,12,5]

Now we can clearly see at a glance that fireBreath worked the way we intended.

Now I want to get really fancy. I want to define a traversal over a geographic area. Can I do that?

Notice how expressive that code is: we want to decrement the health of all units around the target. That code conveys our intention much more clearly than the equivalent mainstream imperative code and it leaves much less room for error.

Anyway, back to breathing fire. First, let's see where the units are located:

I guess people really aren't joking when they say Haskell is the finest imperative language.

Conclusions

This really just scratches the surface of the lens library, which is one of the crown jewels of the Haskell ecosystem. You can use lenses for pure programming, too, and compress very powerful and complex computations into very readable and elegant code. When I have more time I will write even more about this amazing library.

To expand on this, the subexpression (around target 1.0) is an outlaw traversal, but larger subexpression ((around target 1.0).health) is a valid traversal, so in that sense there is nothing wrong with your code. Because the target and health are disjoint lenses, the danger of around has been diffused, and this is almost always how the filter function ends up being used in practice.

There is a lawful way to write this sort of program (i.e. so that all lensy subexpressions are law abiding) but it is so awkward t do that I cannot realistically recommend it. What I personally might do instead is rewrite around so the the code ends up instead as (around target 1.0 health) and while this isn't technically any safer, at least there is a place in the precondition for the around function for me to comment that the two lens arguments must be disjoint lenses.

Hello, Gabriel.When I just started learning Haskell at 2010 there was obvious (to me) that the main library for doing IO right way is the pipes of You (because of its simplicity and beauty).The other main library for working with records of data was/is lens of Edward.Since that year I've read a lot of Haskell books (and papers), but still far away from the full understanding of all the details of the mentioned libs.I don't want to hurry/rush, just want to say that learning Haskell is really challenging and completely refactors the way my mind works solving programming problems.Suddenly I realized, that spent a lot of time before (2010) to gain enouth imperative-experience in order to be able to quickly learn the next imperative language just for a month or so without problems.But, with Haskell the situation is different - it is a complete paradigm shift (I was not given any mention about FP in my University and started learn it at the already mentioned year).Of cause, learning FP brought to my attention a lot of interesting libs for Java/C++/JS which I use at work, but still [almost except Clojure] have no any experience with Haskell.

What I dream of ???At the beginning of learning C++ I gave a wonderful book of Bjarne and found a lot of usefull stuff that changed my mind about programming from it. Then there was other good books, but Stroustroup's one is actual even today.In Haskell the situation is different - the language and ecosystem is much more dynamic/evolving. The really important stuff is usually spread across small articles, blogs, mail-threads.So, my dream here is to have (some day) detailed books about the 2 mentioned libraries, which will be really helpful.Of cause, I know that the best learning about pipes is, probably, your tutorial. I started reading it slowly several month ago, but, by the end - you've already released the next version of the pipes and I'm going to start reading it from scrach again soon.

My questions are:- What can you suggest about lens ?- What is your experience with FP/Haskell, how did you come to doing the stuff you do ?

I plan on writing a Haskell programming book once I finish my PhD so that you won't have to comb over blog posts and mailing threads to learn these things.

The `pipes` API is pretty stable at this point. Most of the research right now is in the derived libraries like `pipes-parse`. The only thing that has really changed since version 3.0 is how to do folds.

Right now there are no real good lens tutorials. That's why I wrote this tutorial to try to fill in the gaps.

My experience in programming in Haskell is as a graduate student in bioinformatics. I actually began programming in C, but my professor kept changing the requirements of my projects every few days, and I could not keep up with him using C because it has really brittle support for data structures and it is very vulnerable to bugs when developing really rapidly.

At that time I sort of understood Haskell a little bit so I decided I wasn't going to get any better until I actually dived in tried to write an entire project using Haskell. So I forced myself to switch to Haskell and rewrote the entire project in Haskell. It took a week to switch the code base over to Haskell, but it was worth it. Once I figured out what I was doing the increase in productivity from using Haskell was very large and now I'm almost done with my degree.

There are several things that greatly increase productivity when programming in Haskell.

First, the strong type system catches mistakes very well. This increases you productivity in multiple ways because:

(A) You spend MUCH less time fixing bugs and maintaining old code

(B) You spend MUCH less time writing tests (the type system is like having lots and lots of unit tests for free)

(C) You can refactor much more easily, which means that you become MUCH braver about exploring alternative approaches to an algorithm or adding new features. It's much easier to say "Yes" to your boss when you have a strong type system because adding new features rarely breaks existing code.

Also, Haskell does not have implicit ANYTHING (i.e. no implicit state, no implicit error handling, no implicit side effects), which makes it very easy to read code and understand existing code. This helps a lot when I have to go back and read my old code after not working on a particular section for a month or two because I need very little context to understand what a given code segment does. Contrast this with other languages where the logic for any feature is spread out all over the code base and it often takes half an hour or more to load all the little bits into your mind's cache before you can begin working on it.

Another feature of programming in Haskell is that it forces you to do things the right way the first time. It strongly discourages you from doing the lazy or hacky thing, so the code you write lasts for a while without any modifications. This is also the reason why people say that Haskell makes you a better programmer, because it forces you to learn the right way of doing everything and you take that with you when you program in other languages. This is true not just for programming style, but also for algorithms and design patterns. Haskell libraries generally feature the state of the art in computer science because the community is top notch, so by programming in Haskell you learn the state of the art very quickly.

Lastly, Haskell permits a very high level of abstraction. When I was translating my C base to Haskell code, I would have C code that was 300 lines long become a 15-liner in Haskell. This makes it significantly easier to reason about the code and find logical mistakes. Generally I find that the strong type system catches all the low-level bugs, and the high-level of abstraction catches all the high-level "business logic" bugs.

My recommendation for you is that if you are interested in Haskell then set aside the time to complete an entire hobby project in Haskell from start to finish. You will learn much faster this way. The moment I actually forced myself to code difficult projects in Haskell was when my proficiency with the language increased very rapidly.

Thank you, Gabriel, very much for the really informative and motivating answer.I clearly understand that there are no other way learning something (Haskell, C++, Java, even car driving) without practice.Of cause, that's not so easy to find a time for doing some real [hobby] project since I'm a full-time software developer and have to do my day-to-day work mostly in C++/JS/Java(with one little utility in Clojure).But, from time to time, there are some small periods of free time, small tasks that could be done not-for-production (like testing data generation using templates, parsing and analyzing logs and so on). Usually I try distinct tools for doing this job (from R-statistics to AWK, batch scrips, Ruby and even Haskell). What I'm planning to do is - to start using Haskell more extensively here in order to "FORM a HABIT".Reading books/articles/blogs about Haskell is already such my habit (and I'm not going to give it up).I even created a small tex/pdf presentation about Haskell fundamentals (motivated by the series of posts about Monads using Kleisli-category by Mike Vanier, who is definitely have a talent of explaining things) in 2011 and presented it 3 times since then (and hope to do it 4-th).But, this is just an exception. I like reading much more than writing and, probably, not going to write more about haskell before understand it better myself.

That is exactly what my teacher of math-analysis told me in 2011. We are still communicating with him weekly even after 10 years since my graduation. That was, frankly speaking, his idea for me to prepare such talk about Haskell and monads. Unfortunately, the fist time I presented it to more than a hundred of his first-second year students, I felt that no one follows me after half an hour. But later, when I gave the same talk to my colleagues at work - everything went well (they were able to keep a track of things).

I was wondering if you knew of any easy way to make new infix functions in the form of (-=). Something for strings would be neat. I've had a look at how (-=) is defined here: ( http://hackage.haskell.org/packages/archive/lens/3.9.0.2/doc/html/src/Control-Lens-Setter.html ) but I got stuck trying to make "State.modify" work.

Yeah, I do. If you can explain what you want your new function to do I can walk you through how you would derive the implementation using lenses. Also, Lens already has a modify-like infix function called `%=` at:

This captures assignment. Then I can make a more natural infix version of this:

xLens +=+ x = xLens %= (first x)

I know (+=+) is a silly thing to call assignment, I've just run out of symbols. Is there some other operator which already does this? It seems like an appropriate thing to have kicking around. Unless there's some problem with this kind of approach that I'm not seeing.

Gabriel, Thanks for this tutorial. I started learning Haskell last year and after about 80 project Euler problems, I thought that Haskell was the greatest language ever. Then I decided to start writing a nosql db in haskell and my enthusiasm was diminishing rapidly until I saw this tutorial which clearly explained lens. Most haskell tutorials I have seen use examples that take as much effort to understand as the concepts they are trying to teach. Your example did not get in the way of the concepts that you were teaching making it one of the best I have seen in the Haskell world. I wish that others would copy you. I intend to if I get good enough to write something usefull.

Let me advise you of one thing: don't wait until you are good enough to start writing tutorials, too. This is the mistake that every Haskell programmer makes and the reason why there are no good Haskell tutorials! :)

You should make a habit of writing from the beginning, because if you don't exercise that skill early on you won't be able to write well when you become expert. The whole reason I got better at tutorials is that I practiced doing so even while I was a beginner. I would just submit everything I wrote to /r/haskell and they would tear it apart week after week. At first I would get defensive and be a bit ashamed, but after doing this over and over again and learning from their criticisms my writing style and skills improved.

Also, writing will help improve your own understanding of the language. Nothing teaches yourself better than teaching others.

This is a really fantastic tutorial. Now I can start using lenses right away without having to have spent hours learning the theory behind them. It's amazing that these imperative-like constructs have been built using purely functional concepts, such as functors. Though it looks like there is some serious Haskell wizardry going on under the covers. If this library catches on, it could deeply impact how we write Haskell, especially when we deal with nested record data types (and perhaps in many other ways, too; I'm still very new to all this).Thank you so much. I really hope you write a book.If you know of similar tutorials that show how to use other combinators, let me know. I'd also like to know if there are other neat tricks in addition to "traversed" and "zoom".

First off, a very great article. I had a lot to learn about this. I am curious however, that how can you use the lift function before defining what it is? i.e. you have't declared (StateT Game) an instance of class MonadTrans t. It is only when you define it as an instance, then you provide the implementation for the lift method. Is this somehow done automatically? Did I miss something?

`StateT` already implements the `MonadTrans` type class. The `transformers` library that we get `StateT` from also provides this `MonadTrans` instance for `StateT`. You can find the code in this module:

My understanding is that you can use lenses for this but you pay an efficiency price when they refer to mutable references. You can use them, but you usually end up having to touch the mutable reference twice for every lens-based operation.

Now I wonder, all your "actions" are of Type Game -> Game, effectively meaning that an action like (fireBreath) is issued by the game world, and not the Boss.

Now I'm curious if it's possible like you would in C++ or similar, to assign specific actions to your units, This means that a Unit has control of itself, and you could effectively isolate off illogical actions, like the players using fireBreath, (or in more complicated situations, like requiring a spell book, before you can cast a spell from it!)

An interesting situation is that in a multiplayer game someone might try to manipulate network packets, to modify what kind of attack the player is issuing, if all attacks are part of the world, its theoretical that a player (without runtime checks) could issue a spell he does not have. (For example, spells would have a global Enum value, instead of the boss and player having their own Enums)

Inspired from your blog post I have ported all your Haskell code into PureScript to get a better understanding of using states and lenses in PureScript: https://github.com/sectore/program-imperatively-using-purescript