Preface

The aims of this book are two-fold: to introduce monads, functors, and other functional programming patterns as a way to structure program design, and to explain how these concepts are implemented in Cats.

Monads, and related concepts, are the functional programming equivalent of object-oriented design patterns—architectural building blocks that turn up over and over again in code. They differ from object-oriented patterns in two main ways:

they are formally, and thus precisely, defined; and

they are extremely (extremely) general.

This generality means they can be difficult to understand. Everyone finds abstraction difficult. However, it is generality that allows concepts like monads to be applied in such a wide variety of situations.

In this book we aim to show the concepts in a number of different ways, to help you build a mental model of how they work and where they are appropriate. We have extended case studies, a simple graphical notation, many smaller examples, and of course the mathematical definitions. Between them we hope you’ll find something that works for you.

Ok, let’s get started!

Versions

This book is written for Scala 2.12.3 and Cats 1.0.0. Here is a minimal build.sbt containing the relevant dependencies and settings1:

Template Projects

For convenience, we have created a Giter8 template to get you started. To clone the template type the following:

$ sbt new underscoreio/cats-seed.g8

This will generate a sandbox project with Cats as a dependency. See the generated README.md for instructions on how to run the sample code and/or start an interactive Scala console.

The cats-seed template is as minimal as it gets. If you’d prefer a more batteries-included starting point, check out Typelevel’s sbt-catalysts template:

$ sbt new typelevel/sbt-catalysts.g8

This will generate a project with a suite of library dependencies and compiler plugins, together with templates for unit tests and tut-enabled documentation. See the project pages for catalysts and sbt-catalysts for more information.

Conventions Used in This Book

This book contains a lot of technical information and program code. We use the following typographical conventions to reduce ambiguity and highlight important concepts:

Typographical Conventions

New terms and phrases are introduced in italics. After their initial introduction they are written in normal roman font.

Terms from program code, filenames, and file contents, are written in monospace font. Note that we do not distinguish between singular and plural forms. For example, we might write String or Strings to refer to java.lang.String.

References to external resources are written as hyperlinks. References to API documentation are written using a combination of hyperlinks and monospace font, for example: scala.Option.

Source Code

Source code blocks are written as follows. Syntax is highlighted appropriately where applicable:

Most code passes through tut to ensure it compiles. tut uses the Scala console behind the scenes, so we sometimes show console-style output as comments:

"Hello Cats!".toUpperCase// res0: String = HELLO CATS!

Callout Boxes

We use two types of callout box to highlight particular content:

Tip callouts indicate handy summaries, recipes, or best practices.

Advanced callouts provide additional information on corner cases or underlying mechanisms. Feel free to skip these on your first read-through—come back to them later for extra information.

Acknowledgements

We’d like to thank our colleagues at Underscore, our friends at Typelevel, and everyone who helped contribute to this book. Special thanks to Jenny Clements for her fantastic artwork and Richard Dallaway for his proof reading expertise. Here is an alphabetical list of contributors:

If you spot an error or potential improvement, please raise an issue or submit a PR on the book’s Github page.

Backers

We’d also like to extend very special thanks to our backers—fine people who helped fund the development of the book by buying a copy before we released it as open source. This book wouldn’t exist without you:

1 Introduction

Cats contains a wide variety of functional programming tools and allows developers to pick and choose the ones we want to use. The majority of these tools are delivered in the form of type classes that we can apply to existing Scala types.

Type classes are a programming pattern originating in Haskell2. They allow us to extend existing libraries with new functionality, without using traditional inheritance, and without altering the original library source code.

In this chapter we will refresh our memory of type classes from Underscore’s Essential Scala book, and take a first look at the Cats codebase. We will look at two example type classes—Show and Eq—using them to identify patterns that lay the foundations for the rest of the book.

1.1 Anatomy of a Type Class

There are three important components to the type class pattern: the type class itself, instances for particular types, and the interface methods that we expose to users.

1.1.1 The Type Class

A type class is an interface or API that represents some functionality we want to implement. In Cats a type class is represented by a trait with at least one type parameter. For example, we can represent generic “serialize to JSON” behaviour as follows:

The compiler spots that we’ve called the toJson method without providing the implicit parameters. It tries to fix this by searching for type class instances of the relevant types and inserting them at the call site:

Json.toJson(Person("Dave", "dave@example.com"))(personWriter)

Interface Syntax

We can alternatively use extension methods to extend existing types with interface methods3. Cats refers to this as “syntax” for the type class:

Most type classes in Cats provide other means to summon instances. However, implicitly is a good fallback for debugging purposes. We can insert a call to implicitly within the general flow of our code to ensure the compiler can find an instance of a type class and ensure that there are no ambiguous implicit errors.

1.2 Working with Implicits

Working with type classes in Scala means working with implicit values and implicit parameters. There are a few rules we need to know to do this effectively.

1.2.1 Packaging Implicits

In a curious quirk of the language, any definitions marked implicit in Scala must be placed inside an object or trait rather than at the top level. In the example above we packaged our type class instances in an object called JsonWriterInstances. We could equally have placed them in a companion object to JsonWriter. Placing instances in a companion object to the type class has special significance in Scala because it plays into something called implicit scope.

1.2.2 Implicit Scope

As we saw above, the compiler searches for candidate type class instances by type. For example, in the following expression it will look for an instance of type JsonWriter[String]:

Json.toJson("A string!")

The compiler searches for candidate instances in the implicit scope at the call site, which roughly consists of:

local or inherited definitions;

imported definitions;

definitions in the companion object of the type class or the parameter type (in this case JsonWriter or String).

Definitions are only included in implicit scope if they are tagged with the implicit keyword. Furthermore, if the compiler sees multiple candidate definitions, it fails with an ambiguous implicit values error:

The precise rules of implicit resolution are more complex than this, but the complexity is largely irrelevant for this book4. For our purposes, we can package type class instances in roughly four ways:

by placing them in an object such as JsonWriterInstances;

by placing them in a trait;

by placing them in the companion object of the type class;

by placing them in the companion object of the parameter type.

With option 1 we bring instances into scope by importing them. With option 2 we bring them into scope with inheritance. With options 3 and 4, instances are always in implicit scope, regardless of where we try to use them.

1.2.3 Recursive Implicit Resolution

The power of type classes and implicits lies in the compiler’s ability to combine implicit definitions when searching for candidate instances.

Earlier we insinuated that all type class instances are implicit vals. This was a simplification. We can actually define instances in two ways:

by defining concrete instances as implicit vals of the required type5;

by defining implicit methods to construct instances from other type class instances.

Why would we construct instances from other instances? As a motivational example, consider defining a JsonWriter for Options. We would need a JsonWriter[Option[A]] for every A we care about in our application. We could try to brute force the problem by creating a library of implicit vals:

This method constructs a JsonWriter for Option[A] by relying on an implicit parameter to fill in the A-specific functionality. When the compiler sees an expression like this:

Json.toJson(Option("A string"))

it searches for an implicit JsonWriter[Option[String]]. It finds the implicit method for JsonWriter[Option[A]]:

Json.toJson(Option("A string"))(optionWriter[String])

and recursively searches for a JsonWriter[String] to use as the parameter to optionWriter:

Json.toJson(Option("A string"))(optionWriter(stringWriter))

In this way, implicit resolution becomes a search through the space of possible combinations of implicit definitions, to find a combination that summons a type class instance of the correct overall type.

Implicit Conversions

When you create a type class instance constructor using an implicit def, be sure to mark the parameters to the method as implicit parameters. Without this keyword, the compiler won’t be able to fill in the parameters during implicit resolution.

implicit methods with non-implicit parameters form a different Scala pattern called an implicit conversion. This is also different from the previous section on Interface Syntax, because in that case the JsonWriter is an implicit class with extension methods. Implicit conversion is an older programming pattern that is frowned upon in modern Scala code. Fortunately, the compiler will warn you when you do this. You have to manually enable implicit conversions by importing scala.language.implicitConversions in your file:

implicitdef optionWriter[A]
(writer: JsonWriter[A]): JsonWriter[Option[A]] =
???
// <console>:18: warning: implicit conversion method optionWriter should be enabled// by making the implicit value scala.language.implicitConversions visible.// This can be achieved by adding the import clause 'import scala.language.implicitConversions'// or by setting the compiler option -language:implicitConversions.// See the Scaladoc for value scala.language.implicitConversions for a discussion// why the feature should be explicitly enabled.// implicit def optionWriter[A]// ^// error: No warnings can be incurred under -Xfatal-warnings.

1.3 Exercise: Printable Library

Scala provides a toString method to let us convert any value to a String. However, this method comes with a few disadvantages: it is implemented for every type in the language, many implementations are of limited use, and we can’t opt-in to specific implementations for specific types.

Let’s define a Printable type class to work around these problems:

Define a type class Printable[A] containing a single method format. format should accept a value of type A and return a String.

Create an object PrintableInstances containing instances of Printable for String and Int.

Define an object Printable with two generic interface methods:

format accepts a value of type A and a Printable of the corresponding type. It uses the relevant Printable to convert the A to a String.

print accepts the same parameters as format and returns Unit. It prints the A value to the console using println.

These steps define the three main components of our type class. First we define Printable—the type class itself:

trait Printable[A] {
defformat(value: A): String
}

Then we define some default instances of Printable and package them in PrintableInstances:

Finally, we use the type class by bringing the relevant instances into scope and using interface object/syntax. If we defined the instances in companion objects Scala brings them into scope for us automatically. Otherwise we use an import to access them:

1.4 Meet Cats

In the previous section we saw how to implement type classes in Scala. In this section we will look at how type classes are implemented in Cats.

Cats is written using a modular structure that allows us to choose which type classes, instances, and interface methods we want to use. Let’s take a first look using cats.Show as an example.

Show is Cats’ equivalent of the Printable type class we defined in the last section. It provides a mechanism for producing developer-friendly console output without using toString. Here’s an abbreviated definition:

package cats
trait Show[A] {
defshow(value: A): String
}

1.4.1 Importing Type Classes

The type classes in Cats are defined in the cats package. We can import Show directly from this package:

import cats.Show

The companion object of every Cats type class has an apply method that locates an instance for any type we specify:

Oops—that didn’t work! The apply method uses implicits to look up individual instances, so we’ll have to bring some instances into scope.

1.4.2 Importing Default Instances

The cats.instances package provides default instances for a wide variety of types. We can import these as shown in the table below. Each import provides instances of all Cats’ type classes for a specific parameter type:

Cats provides separate syntax imports for each type class. We will introduce these as we encounter them in later sections and chapters.

1.4.4 Importing All The Things!

In this book we will use specific imports to show you exactly which instances and syntax you need in each example. However, this can be time consuming for many use cases. You should feel free to take one of the following shortcuts to simplify your imports:

import cats._ imports all of Cats’ type classes in one go;

import cats.instances.all._ imports all of the type class instances for the standard library in one go;

import cats.syntax.all._ imports all of the syntax in one go;

import cats.implicits._ imports all of the standard type class instances and all of the syntax in one go.

Most people start their files with the following imports, reverting to more specific imports only if they encounter naming conflicts or problems ambiguous implicits:

import cats._
import cats.implicits._

1.4.5 Defining Custom Instances

We can define an instance of Show simply by implementing the trait for a given type:

However, Cats also provides a couple of convenient methods to simplify the process. There are two construction methods on the companion object of Show that we can use to define instances for our own types:

As you can see, the code using construction methods is much terser than the code without. Many type classes in Cats provide helper methods like these for constructing instances, either from scratch or by transforming existing instances for other types.

1.4.6 Exercise: Cat Show

Re-implement the Cat application from the previous section using Show instead of Printable.

First let’s import everything we need from Cats: the Show type class, the instances for Int and String, and the interface syntax:

Ok, many of you won’t have made such a simple mistake as this, but the principle is sound. The predicate in the filter clause always returns false because it is comparing an Int to an Option[Int].

This is programmer error—we should have compared item to Some(1) instead of 1. However, it’s not technically a type error because == works for any pair of objects, no matter what types we compare. Eq is designed to add some type safety to equality checks and work around this problem.

1.5.1 Equality, Liberty, and Fraternity

We can use Eq to define type-safe equality between instances of any given type:

We have received an error here because the types don’t quite match up. We have Eq instances in scope for Int and Option[Int] but the values we are comparing are of type Some[Int]. To fix the issue we have to re-type the arguments as Option[Int]:

1.6 Controlling Instance Selection

When working with type classes we must consider two issues that control instance selection:

What is the relationship between an instance defined on a type and its subtypes?

For example, if we define a JsonWriter[Option[Int]], will the expression Json.toJson(Some(1)) select this instance? (Remember that Some is a subtype of Option).

How do we choose between type class instances when there are many available?

What if we define two JsonWriters for Person? When we write Json.toJson(aPerson), which instance is selected?

1.6.1 Variance

When we define type classes we can add variance annotations to the type parameter to affect the variance of the type class and the compiler’s ability to select instances during implicit resolution.

To recap Essential Scala, variance relates to subtypes. We say that B is a subtype of A if we can use a value of type B anywhere we expect a value of type A.

Co- and contravariance annotations arise when working with type constructors. For example, we denote covariance with a + symbol:

trait F[+A] // the "+" means "covariant"

Covariance

Covariance means that the type F[B] is a subtype of the type F[A] if B is a subtype of A. This is useful for modelling many types, including collections like List and Option:

trait List[+A]
trait Option[+A]

The covariance of Scala collections allows us to substitute collections of one type for another in our code. For example, we can use a List[Circle] anywhere we expect a List[Shape] because Circle is a subtype of Shape:

Let’s unpack this a bit further. Remember that variance is all about the ability to substitute one value for another. Consider a scenario where we have two values, one of type Shape and one of type Circle, and two JsonWriters, one for Shape and one for Circle:

Now ask yourself the question: “Which of combinations of value and writer can I pass to format?” We can combine circle with either writer because all Circles are Shapes. Conversely, we can’t combine shape with circleWriter because not all Shapes are Circles.

This relationship is what we formally model using contravariance. JsonWriter[Shape] is a subtype of JsonWriter[Circle] because Circle is a subtype of Shape. This means we can use shapeWriter anywhere we expect to see a JsonWriter[Circle].

Invariance

Invariance is actually the easiest situation to describe. It’s what we get when we don’t write a + or - in a type constructor:

trait F[A]

This means the types F[A] and F[B] are never subtypes of one another, no matter what the relationship between A and B. This is the default semantics for Scala type constructors.

When the compiler searches for an implicit it looks for one matching the type or subtype. Thus we can use variance annotations to control type class instance selection to some extent.

There are two issues that tend to arise. Let’s imagine we have an algebraic data type like:

sealedtrait A
finalcaseobject B extends A
finalcaseobject C extends A

The issues are:

Will an instance defined on a supertype be selected if one is available? For example, can we define an instance for A and have it work for values of type B and C?

Will an instance for a subtype be selected in preference to that of a supertype. For instance, if we define an instance for A and B, and we have a value of type B, will the instance for B be selected in preference to A?

It turns out we can’t have both at once. The three choices give us behaviour as follows:

Type Class Variance

Invariant

Covariant

Contravariant

Supertype instance used?

No

No

Yes

More specific type preferred?

No

Yes

No

It’s clear there is no perfect system. Cats generally prefers to use invariant type classes. This allows us to specify more specific instances for subtypes if we want. It does mean that if we have, for example, a value of type Some[Int], our type class instance for Option will not be used. We can solve this problem with a type annotation like Some(1) : Option[Int] or by using “smart constructors” like the Option.apply, Option.empty, some, and none methods we saw in Section 1.5.3.

1.7 Summary

In this chapter we took a first look at type classes. We implemented our own Printable type class using plain Scala before looking at two examples from Cats—Show and Eq.

In the remaining chapters of Part I we will look at several broad and powerful type classes—Semigroup, Monoid, Functor, Monad, Semigroupal, Applicative, Traverse, and more. In each case we will learn what functionality the type class provides, the formal rules it follows, and how it is implemented in Cats. Many of these type classes are more abstract than Show or Eq. While this makes them harder to learn, it makes them far more useful for solving general problems in our code.

2 Monoids and Semigroups

In this section we explore our first type classes, monoid and semigroup. These allow us to add or combine values. There are instances for Ints, Strings, Lists, Options, and many more. Let’s start by looking at a few simple types and operations to see what common principles we can extract.

Integer addition

Addition of Ints is a binary operation that is closed, meaning that adding two Ints always produces another Int:

2 + 1// res0: Int = 3

There is also the identity element 0 with the property that a + 0 == 0 + a == a for any Inta:

2 + 0// res1: Int = 20 + 2// res2: Int = 2

There are also other properties of addition. For instance, it doesn’t matter in what order we add elements because we always get the same result. This is a property known as associativity:

(1 + 2) + 3// res3: Int = 61 + (2 + 3)
// res4: Int = 6

Integer multiplication

The same properties for addition also apply for multiplication, provided we use 1 as the identity instead of 0:

1 * 3// res5: Int = 33 * 1// res6: Int = 3

Multiplication, like addition, is associative:

(1 * 2) * 3// res7: Int = 61 * (2 * 3)
// res8: Int = 6

String and sequence concatenation

We can also add Strings, using string concatenation as our binary operator:

Note that we used ++ above instead of the more usual + to suggest a parallel with sequences. We can do the same with other types of sequence, using concatenation as the binary operator and the empty sequence as our identity.

2.1 Definition of a Monoid

We’ve seen a number of “addition” scenarios above each with an associative binary addition and an identity element. It will be no surprise to learn that this is a monoid. Formally, a monoid for a type A is:

an operation combine with type (A, A) => A

an element empty of type A

This definition translates nicely into Scala code. Here is a simplified version of the definition from Cats:

trait Monoid[A] {
defcombine(x: A, y: A): A
def empty: A
}

In addition to providing the combine and empty operations, monoids must formally obey several laws. For all values x, y, and z, in A, combine must be associative and empty must be an identity element:

Integer subtraction, for example, is not a monoid because subtraction is not associative:

(1 - 2) - 3// res15: Int = -41 - (2 - 3)
// res16: Int = 2

In practice we only need to think about laws when we are writing our own Monoid instances. Unlawful instances are dangerous because they can yield unpredictable results when used with the rest of Cats’ machinery. Most of the time we can rely on the instances provided by Cats and assume the library authors know what they’re doing.

2.2 Definition of a Semigroup

A semigroup is just the combine part of a monoid. While many semigroups are also monoids, there are some data types for which we cannot define an empty element. For example, we have just seen that sequence concatenation and integer addition are monoids. However, if we restrict ourselves to non-empty sequences and positive integers, we are no longer able to define a sensible empty element. Cats has a NonEmptyList data type that has an implementation of Semigroup but no implementation of Monoid.

A more accurate (though still simplified) definition of Cats’ Monoid is:

We’ll see this kind of inheritance often when discussing type classes. It provides modularity and allows us to re-use behaviour. If we define a Monoid for a type A, we get a Semigroup for free. Similarly, if a method requires a parameter of type Semigroup[B], we can pass a Monoid[B] instead.

2.3 Exercise: The Truth About Monoids

We’ve seen a few examples of monoids but there are plenty more to be found. Consider Boolean. How many monoids can you define for this type? For each monoid, define the combine and empty operations and convince yourself that the monoid laws hold. Use the following definitions as a starting point:

We need to define setUnionMonoid as a method rather than a value so we can accept the type parameter A. The type parameter allows us to use the same definition to summon Monoids for Sets of any type of data:

Set complement and set difference are not associative, so they cannot be considered for either monoids or semigroups. However, symmetric difference (the union less the intersection) does also form a monoid with the empty set:

2.5 Monoids in Cats

Now we’ve seen what monoids are, let’s look at their implementation in Cats. Once again we’ll look at the three main aspects of the implementation: the type class, the instances, and the interface.

2.5.1 The Monoid Type Class

The monoid type class is cats.kernel.Monoid, which is aliased as cats.Monoid. Monoid extends cats.kernel.Semigroup, which is aliased as cats.Semigroup. When using Cats we normally import type classes from the cats package:

import cats.Monoidimport cats.Semigroup

Cats Kernel?

Cats Kernel is a subproject of Cats providing a small set of typeclasses for libraries that don’t require the full Cats toolbox. While these core type classes are technically defined in the cats.kernel package, they are all aliased to the cats package so we rarely need to be aware of the distinction.

The Cats Kernel type classes covered in this book are Eq, Semigroup, and Monoid. All the other type classes we cover are part of the main Cats project and are defined directly in the cats package.

2.5.2 Monoid Instances

Monoid follows the standard Cats pattern for the user interface: the companion object has an apply method that returns the type class instance for a particular type. For example, if we want the monoid instance for String, and we have the correct implicits in scope, we can write the following:

The type class instances for Monoid are organised under cats.instances in the standard way described in Chapter 1. For example, if we want to pull in instances for Int we import from cats.instances.int:

2.5.4 Exercise: Adding All The Things

The cutting edge SuperAdder v3.5a-32 is the world’s first choice for adding together numbers. The main function in the program has signature def add(items: List[Int]): Int. In a tragic accident this code is deleted! Rewrite the method and save the day!

We can write the addition as a simple foldLeft using 0 and the + operator:

defadd(items: List[Int]): Int =
items.foldLeft(0)(_ + _)

We can alternatively write the fold using Monoids, although there’s not a compelling use case for this yet:

Well done! SuperAdder’s market share continues to grow, and now there is demand for additional functionality. People now want to add List[Option[Int]]. Change add so this is possible. The SuperAdder code base is of the highest quality, so make sure there is no code duplication!

Now there is a use case for Monoids. We need a single method that adds Ints and instances of Option[Int]. We can write this as a generic method that accepts an implicit Monoid as a parameter:

2.6 Applications of Monoids

We now know what a monoid is—an abstraction of the concept of adding or combining—but where is it useful? Here are a few big ideas where monoids play a major role. These are explored in more detail in case studies later in the book.

2.6.1 Big Data

In big data applications like Spark and Hadoop we distribute data analysis over many machines, giving fault tolerance and scalability. This means each machine will return results over a portion of the data, and we must then combine these results to get our final result. In the vast majority of cases this can be viewed as a monoid.

If we want to calculate how many total visitors a web site has received, that means calculating an Int on each portion of the data. We know the monoid instance of Int is addition, which is the right way to combine partial results.

If we want to find out how many unique visitors a website has received, that’s equivalent to building a Set[User] on each portion of the data. We know the monoid instance for Set is the set union, which is the right way to combine partial results.

If we want to calculate 99% and 95% response times from our server logs, we can use a data structure called a QTree for which there is a monoid.

Hopefully you get the idea. Almost every analysis that we might want to do over a large data set is a monoid, and therefore we can build an expressive and powerful analytics system around this idea. This is exactly what Twitter’s Algebird and Summingbird projects have done. We explore this idea further in the map-reduce case study.

2.6.2 Distributed Systems

In a distributed system, different machines may end up with different views of data. For example, one machine may receive an update that other machines did not receive. We would like to reconcile these different views, so every machine has the same data if no more updates arrive. This is called eventual consistency.

A particular class of data types support this reconciliation. These data types are called commutative replicated data types (CRDTs). The key operation is the ability to merge two data instances, with a result that captures all the information in both instances. This operation relies on having a monoid instance. We explore this idea further in the CRDT case study.

2.6.3 Monoids in the Small

The two examples above are cases where monoids inform the entire system architecture. There are also many cases where having a monoid around makes it easier to write a small code fragment. We’ll see lots of examples in the case studies in this book.

2.7 Summary

We hit a big milestone in this chapter—we covered our first type classes with fancy functional programming names:

a Semigroup represents an addition or combination operation;

a Monoid extends a Semigroup by adding an identity or “zero” element.

We can use Semigroups and Monoids by importing three things: the type classes themselves, the instances for the types we care about, and the semigroup syntax to give us the |+| operator:

Monoids are a great gateway to Cats. They’re easy to understand and simple to use. However, they’re just the tip of the iceberg in terms of the abstractions Cats enables us to make. In the next chapter we’ll look at functors, the type class personification of the beloved map method. That’s where the fun really begins!

3 Functors

In this chapter we will investigate functors, an abstraction that allows us to represent sequences of operations within a context such as a List, an Option, or any one of a thousand other possibilities. Functors on their own aren’t so useful, but special cases of functors such as monads and applicative functors are some of the most commonly used abstractions in Cats.

3.1 Examples of Functors

Informally, a functor is anything with a map method. You probably know lots of types that have this: Option, List, and Either, to name a few.

We typically first encounter map when iterating over Lists. However, to understand functors we need to think of the method in another way. Rather than traversing the list, we should think of it as transforming all of the values inside in one go. We specify the function to apply, and map ensures it is applied to every item. The values change but the structure of the list remains the same:

List(1, 2, 3).map(n => n + 1)
// res0: List[Int] = List(2, 3, 4)

Similarly, when we map over an Option, we transform the contents but leave the Some or None context unchanged. The same principle applies to Either with its Left and Right contexts. This general notion of transformation, along with the common pattern of type signatures shown in Figure 1, is what connects the behaviour of map across different data types.

Figure 1: Type chart: mapping over List, Option, and Either

Because map leaves the structure of the context unchanged, we can call it repeatedly to sequence multiple computations on the contents of an initial data structure:

We should think of map not as an iteration pattern, but as a way of sequencing computations on values ignoring some complication dictated by the relevant data type:

Option—the value may or may not be present;

Either—there may be a value or an error;

List—there may be zero or more values.

3.2 More Examples of Functors

The map methods of List, Option, and Either apply functions eagerly. However, the idea of sequencing computations is more general than this. Let’s investigate the behaviour of some other functors that apply the pattern in different ways.

Futures

Future is a functor that sequences asynchronous computations by queueing them and applying them as their predecessors complete. The type signature of its map method, shown in Figure 2, has the same shape as the signatures above. However, the behaviour is very different.

Figure 2: Type chart: mapping over a Future

When we work with a Future we have no guarantees about its internal state. The wrapped computation may be ongoing, complete, or rejected. If the Future is complete, our mapping function can be called immediately. If not, some underlying thread pool queues the function call and comes back to it later. We don’t know when our functions will be called, but we do know what order they will be called in. In this way, Future provides the same sequencing behaviour seen in List, Option, and Either:

Note that Scala’s Futures aren’t a great example of pure functional programming because they aren’t referentially transparent. Future always computes and caches a result and there’s no way for us to tweak this behaviour. This means we can get unpredictable results when we use Future to wrap side-effecting computations. For example:

Ideally we would like result1 and result2 to contain the same value. However, the computation for future1 calls nextInt once and the computation for future2 calls it twice. Because nextInt returns a different result every time we get a different result in each case.

This kind of discrepancy makes it hard to reason about programs involving Futures and side-effects. There also are other problematic aspects of Future's behaviour, such as the way it always starts computations immediately rather than allowing the user to dictate when the program should run. For more information see this excellent Reddit answer by Rob Norris.

If Future isn’t referentially transparent, perhaps we should look at another similar data-type that is. You should recognise this one…

Functions (?!)

It turns out that single argument functions are also functors. To see this we have to tweak the types a little. A function A => B has two type parameters: the parameter type A and the result type B. To coerce them to the correct shape we can fix the parameter type and let the result type vary:

start with X => A;

supply a function A => B;

get back X => B.

If we alias X => A as MyFunc[A], we see the same pattern of types we saw with the other examples in this chapter. We also see this in Figure 3:

How does this relate to our general pattern of sequencing operations? If we think about it, function composition is sequencing. We start with a function that performs a single operation and every time we use map we append another operation to the chain. Calling map doesn’t actually run any of the operations, but if we can pass an argument to the final function all of the operations are run in sequence. We can think of this as lazily queueing up operations similar to Future:

For the above examples to work we need to add the following compiler option to build.sbt:

scalacOptions += "-Ypartial-unification"

otherwise we’ll get a compiler error:

func1.map(func2)
// <console>: error: value map is not a member of Int => Double// func1.map(func2)
^

We’ll look at why this happens in detail in Section 3.8.

3.3 Definition of a Functor

Every example we’ve looked at so far is a functor: a class that encapsulates sequencing computations. Formally, a functor is a type F[A] with an operation map with type (A => B) => F[B]. The general type chart is shown in Figure 4.

Figure 4: Type chart: generalised functor map

Cats encodes Functor as a type class, cats.Functor, so the method looks a little different. It accepts the initial F[A] as a parameter alongside the transformation function. Here’s a simplified version of the definition:

If you haven’t seen syntax like F[_] before, it’s time to take a brief detour to discuss type constructors and higher kinded types. We’ll explain that scala.language import as well.

Functor Laws

Functors guarantee the same semantics whether we sequence many small operations one by one, or combine them into a larger function before mapping. To ensure this is the case the following laws must hold:

Identity: calling map with the identity function is the same as doing nothing:

fa.map(a => a) == fa

Composition: mapping with two functions f and g is the same as mapping with f and then mapping with g:

fa.map(g(f(_))) == fa.map(f).map(g)

3.4 Aside: Higher Kinds and Type Constructors

Kinds are like types for types. They describe the number of “holes” in a type. We distinguish between regular types that have no holes and “type constructors” that have holes we can fill to produce types.

For example, List is a type constructor with one hole. We fill that hole by specifying a parameter to produce a regular type like List[Int] or List[A]. The trick is not to confuse type constructors with generic types. List is a type constructor, List[A] is a type:

Armed with this knowledge of type constructors, we can see that the Cats definition of Functor allows us to create instances for any single-parameter type constructor, such as List, Option, Future, or a type alias such as MyFunc.

Language Feature Imports

Higher kinded types are considered an advanced language feature in Scala. Whenever we declare a type constructor with A[_] syntax, we need to “enable” the higher kinded type language feature to suppress warnings from the compiler. We can either do this with a “language import” as above:

import scala.language.higherKinds

or by adding the following to scalacOptions in build.sbt:

scalacOptions += "-language:higherKinds"

We’ll use the language import in this book to ensure we are as explicit as possible. In practice, however, we find the scalacOptions flag to be simpler and less verbose.

3.5 Functors in Cats

Let’s look at the implementation of functors in Cats. We’ll examine the aspects we did for monoids: the type class, the instances, and the syntax.

3.5.1 The Functor Type Class

The functor type class is cats.Functor. We obtain instances using the standard Functor.apply method on the companion object. As usual, default instances are arranged by type in the cats.instances package:

3.5.2 Functor Syntax

The main method provided by the syntax for Functor is map. It’s difficult to demonstrate this with Options and Lists as they have their own built-in map methods and the Scala compiler will always prefer a built-in method over an extension method. We’ll work around this with two examples.

First let’s look at mapping over functions. Scala’s Function1 type doesn’t have a map method (it’s called andThen instead) so there are no naming conflicts:

Let’s look at another example. This time we’ll abstract over functors so we’re not working with any particular concrete type. We can write a method that applies an equation to a number no matter what functor context it’s in:

3.5.3 Instances for Custom Types

We can define a functor simply by defining its map method. Here’s an example of a Functor for Option, even though such a thing already exists in cats.instances. The implementation is trivial—we simply call Option'smap method:

Sometimes we need to inject dependencies into our instances. For example, if we had to define a custom Functor for Future (another hypothetical example—Cats provides one in cats.instances.future) we would need to account for the implicit ExecutionContext parameter on future.map. We can’t add extra parameters to functor.map so we have to account for the dependency when we create the instance:

Whenever we summon a Functor for Future, either directly using Functor.apply or indirectly via the map extension method, the compiler will locate futureFunctor by implicit resolution and recursively search for an ExecutionContext at the call site. This is what the expansion might look like:

// We write this:
Functor[Future]
// The compiler expands to this first:
Functor[Future](futureFunctor)
// And then to this:
Functor[Future](futureFunctor(executionContext))

3.5.4 Exercise: Branching out with Functors

Write a Functor for the following binary tree data type. Verify that the code works as expected on instances of Branch and Leaf:

The semantics are similar to writing a Functor for List. We recurse over the data structure, applying the function to every Leaf we find. The functor laws intuitively require us to retain the same structure with the same pattern of Branch and Leaf nodes:

Oops! This falls foul of the same invariance problem we discussed in Section 1.6.1. The compiler can find a Functor instance for Tree but not for Branch or Leaf. Let’s add some smart constructors to compensate:

3.6Contravariant and Invariant Functors

As we have seen, we can think of Functor'smap method as “appending” a transformation to a chain. We’re now going to look at two other type classes, one representing prepending operations to a chain, and one representing building a bidirectional chain of operations. These are called contravariant and invariant functors respectively.

This Section is Optional!

You don’t need to know about contravariant and invariant functors to understand monads, which are the most important pattern in this book and the focus of the next chapter. However, contravariant and invariant do come in handy in our discussion of Semigroupal and Applicative in Chapter 6.

If you want to move on to monads now, feel free to skip straight to Chapter 4. Come back here before you read Chapter 6.

3.6.1 Contravariant Functors and the contramap Method

The first of our type classes, the contravariant functor, provides an operation called contramap that represents “prepending” an operation to a chain. The general type signature is shown in Figure 5.

Figure 5: Type chart: the contramap method

The contramap method only makes sense for data types that represent transformations. For example, we can’t define contramap for an Option because there is no way of feeding a value in an Option[B] backwards through a function A => B. However, we can define contramap for the Printable type class we discussed in Chapter 1:

trait Printable[A] {
defformat(value: A): String
}

A Printable[A] represents a transformation from A to String. Its contramap method accepts a function func of type B => A and creates a new Printable[B]:

If you get stuck, think about the types. You need to turn value, which is of type B, into a String. What functions and methods do you have available and in what order do they need to be combined?

Here’s a working implementation. We call func to turn the B into an A and then use our original Printable to turn the A into a String. In a small show of sleight of hand we use a self alias to distinguish the outer and inner Printables:

3.6.2 Invariant functors and the imap method

Invariant functors implement a method called imap that is informally equivalent to a combination of map and contramap. If map generates new type class instances by appending a function to a chain, and contramap generates them by prepending an operation to a chain, imap generates them via a pair of bidirectional transformations.

The most intuitive examples of this are a type class that represents encoding and decoding as some data type, such as Play JSON’s Format and scodec’s Codec. We can build our own Codec by enhancing Printable to support encoding and decoding to/from a String:

What’s the relationship between the terms “contravariance”, “invariance”, and “covariance” and these different kinds of functor?

If you recall from Section 1.6.1, variance affects subtyping, which is essentially our ability to use a value of one type in place of a value of another type without breaking the code.

Subtyping can be viewed as a conversion. If B is a subtype of A, we can always convert a B to an A.

Equivalently we could say that B is a subtype of A if there exists a function A => B. A standard covariant functor captures exactly this. If F is a covariant functor, wherever we have an F[A] and a conversion A => B we can always convert to an F[B].

A contravariant functor captures the opposite case. If F is a contravariant functor, whenever we have a F[A] and a conversion B => A we can convert to an F[B].

Finally, invariant functors capture the case where we can convert from F[A] to F[B] via a function A => B and vice versa via a function B => A.

3.7Contravariant and Invariant in Cats

Let’s look at the implementation of contravariant and invariant functors in Cats, provided by the cats.Contravariant and cats.Invariant type classes. Here’s a simplified version of the code:

Imagine we want to produce a Monoid for Scala’s Symbol type. Cats doesn’t provide a Monoid for Symbol but it does provide a Monoid for a similar type: String. We can write our new semigroup with an empty method that relies on the empty String, and a combine method that works as follows:

accept two Symbols as parameters;

convert the Symbols to Strings;

combine the Strings using Monoid[String];

convert the result back to a Symbol.

We can implement combine using imap, passing functions of type String => Symbol and Symbol => String as parameters. Here’ the code, written out using the imap extension method provided by cats.syntax.invariant:

Obviously “partial unification” is some kind of optional compiler behaviour, without which our code will not compile. We should take a moment to describe this behaviour and discuss some gotchas and workarounds.

3.8.1 Unifying Type Constructors

In order to compile an expression like func1.map(func2) above, the compiler has to search for a Functor for Function1. However, Functor accepts a type constructor with one parameter:

trait Functor[F[_]] {
def map[A, B](fa: F[A])(func: A => B): F[B]
}

and Function1 has two type parameters (the function argument and the result type):

trait Function1[-A, +B] {
defapply(arg: A): B
}

The compiler has to fix one of the two parameters of Function1 to create a type constructor of the correct kind to pass to Functor. It has two options to choose from:

type F[A] = Int => A
type F[A] = A => Double

We know that the former of these is the correct choice. However, earlier versions of the Scala compiler were not able to make this inference. This infamous limitation, known as SI-2712, prevented the compiler from “unifying” type constructors of different arities. This compiler limitation is now fixed, although we have to enable the fix via a compiler flag in build.sbt:

scalacOptions += "-Ypartial-unification"

3.8.2 Left-to-Right Elimination

The partial unification in the Scala compiler works by fixing type parameters from left to right. In the above example, the compiler fixes the Int in Int => Double and looks for a Functor for functions of type Int => ?:

type F[A] = Int => A
val functor = Functor[F]

This left-to-right elimination works for a wide variety of common scenarios, including Functors for types such as Function1 and Either:

However, there are situations where left-to-right elimination is not the correct choice. One example is the Or type in Scalactic, which is a conventionally left-biased equivalent of Either:

type PossibleResult = ActualResult Or Error

Another example is the Contravariant functor for Function1.

While the covariant Functor for Function1 implements andThen-style left-to-right function composition, the Contravariant functor implements compose-style right-to-left composition. In other words, the following expressions are all equivalent:

The problem here is that the Contravariant for Function1 fixes the return type and leaves the parameter type varying, requiring the compiler to eliminate type parameters from right to left, as shown below and in Figure 7:

type F[A] = A => Double

Figure 7: Type chart: contramapping over a Function1

The compiler fails simply because of its left-to-right bias. We can prove this by creating a type alias that flips the parameters on Function1:

type <=[B, A] = A => B
type F[A] = Double <= A

If we re-type func2 as an instance of <=, we reset the required order of elimination and we can call contramap as desired:

The difference between func2 and func2b is purely syntactic—both refer to the same value and the type aliases are otherwise completely compatible. Incredibly, however, this simple rephrasing is enough to give the compiler the hint it needs to solve the problem.

It is rare that we have to do this kind of right-to-left elimination. Most multi-parameter type constructors are designed to be right-biased, requiring the left-to-right elimination that is supported by the compiler out of the box. However, it is useful to know about -Ypartial-unification and this quirk of elimination order in case you ever come across an odd scenario like the one above.

3.9 Summary

Functors represent sequencing behaviours. We covered three types of functor in this chapter:

Regular covariant Functors, with their map method, represent the ability to apply functions to a value in some context. Successive calls to map apply these functions in sequence, each accepting the result of its predecessor as a parameter.

Contravariant functors, with their contramap method, represent the ability to “prepend” functions to a function-like context. Successive calls to contramap sequence these functions in the opposite order to map.

Regular Functors are by far the most common of these type classes, but even then it is rare to use them on their own. Functors form a foundational building block of several more interesting abstractions that we use all the time. In the following chapters we will look at two of these abstractions: monads and applicative functors.

Functors for collections are extremely important, as they transform each element independently of the rest. This allows us to parallelise or distribute transformations on large collections, a technique leveraged heavily in “map-reduce” frameworks like Hadoop. We will investigate this approach in more detail in the Map-reduce case study later in the book.

The Contravariant and Invariant type classes are less widely applicable but are still useful for building data types that represent transformations. We will revisit them to discuss the Semigroupal type class later in Chapter 6.

4 Monads

Monads are one of the most common abstractions in Scala. Many Scala programmers quickly become intuitively familiar with monads, even if we don’t know them by name.

Informally, a monad is anything with a constructor and a flatMap method. All of the functors we saw in the last chapter are also monads, including Option, List, and Future. We even have special syntax to support monads: for comprehensions. However, despite the ubiquity of the concept, the Scala standard library lacks a concrete type to encompass “things that can be flatMapped”. This type class is one of the benefits brought to us by Cats.

In this chapter we will take a deep dive into monads. We will start by motivating them with a few examples. We’ll proceed to their formal definition and their implementation in Cats. Finally, we’ll tour some interesting monads that you may not have seen, providing introductions and examples of their use.

4.1 What is a Monad?

This is the question that has been posed in a thousand blog posts, with explanations and analogies involving concepts as diverse as cats, Mexican food, space suits full of toxic waste, and monoids in the category of endofunctors (whatever that means). We’re going to solve the problem of explaining monads once and for all by stating very simply:

A monad is a mechanism for sequencing computations.

That was easy! Problem solved, right? But then again, last chapter we said functors were a control mechanism for exactly the same thing. Ok, maybe we need some more discussion…

In Section 3.1 we said that functors allow us to sequence computations ignoring some complication. However, functors are limited in that they only allow this complication to occur once at the beginning of the sequence. They don’t account further complications at each step in the sequence.

This is where monads come in. A monad’s flatMap method allows us to specify what happens next, taking into account an intermediate complication. The flatMap method of Option takes intermediate Options into account. The flatMap method of List handles intermediate Lists. And so on. In each case, the function passed to flatMap specifies the application-specific part of the computation, and flatMap itself takes care of the complication allowing us to flatMap again. Let’s ground things by looking at some examples.

Options

Option allows us to sequence computations that may or may not return values. Here are some examples:

if it returns a Some, the flatMap method calls our function and passes us the integer aNum;

the second call to parseInt returns a None or a Some;

if it returns a Some, the flatMap method calls our function and passes us bNum;

the call to divide returns a None or a Some, which is our result.

At each step, flatMap chooses whether to call our function, and our function generates the next computation in the sequence. This is shown in Figure 8.

Figure 8: Type chart: flatMap for Option

The result of the computation is an Option, allowing us to call flatMap again and so the sequence continues. This results in the fail-fast error handling behaviour that we know and love, where a None at any step results in a None overall:

Every monad is also a functor (see below for proof), so we can rely on both flatMap and map to sequence computations that do and don’t introduce a new monad. Plus, if we have both flatMap and map we can use for comprehensions to clarify the sequencing behaviour:

When we first encounter flatMap as budding Scala developers, we tend to think of it as a pattern for iterating over Lists. This is reinforced by the syntax of for comprehensions, which look very much like imperative for loops:

However, there is another mental model we can apply that highlights the monadic behaviour of List. If we think of Lists as sets of intermediate results, flatMap becomes a construct that calculates permutations and combinations.

For example, in the for comprehension above there are three possible values of x and two possible values of y. This means there are six possible values of (x, y). flatMap is generating these combinations from our code, which states the sequence of operations:

get x

get y

create a tuple (x, y)

Futures

Future is a monad that sequences computations without worrying that they are asynchronous:

Again, we specify the code to run at each step, and flatMap takes care of all the horrifying underlying complexities of thread pools and schedulers.

If you’ve made extensive use of Future, you’ll know that the code above is running each operation in sequence. This becomes clearer if we expand out the for comprehension to show the nested calls to flatMap:

Each Future in our sequence is created by a function that receives the result from a previous Future. In other words, each step in our computation can only start once the previous step is finished. This is born out by the type chart for flatMap in Figure 9, which shows the function parameter of type A => Future[B].

Figure 9: Type chart: flatMap for Future

We can run futures in parallel, of course, but that is another story and shall be told another time. Monads are all about sequencing.

4.1.1 Definition of a Monad

While we have only talked about flatMap above, monadic behaviour is formally captured in two operations:

pure abstracts over constructors, providing a way to create a new monadic context from a plain value. flatMap provides the sequencing step we have already discussed, extracting the value from a context and generating the next context in the sequence. Here is a simplified version of the Monad type class in Cats:

At first glance this seems tricky, but if we follow the types we’ll see there’s only one solution. We are passed a value of type F[A]. Given the tools available there’s only one thing we can do: call flatMap:

We need a function of type A => F[B] as the second parameter. We have two function building blocks available: the func parameter of type A => B and the pure function of type A => F[A]. Combining these gives us our result:

4.2 Monads in Cats

It’s time to give monads our standard Cats treatment. As usual we’ll look at the type class, instances, and syntax.

4.2.1 The Monad Type Class

The monad type class is cats.Monad. Monad extends two other type classes: FlatMap, which provides the flatMap method, and Applicative, which provides pure. Applicative also extends Functor, which gives every Monad a map method as we saw in the exercise above. We’ll discuss Applicatives in Chapter 6.

Cats also provides a Monad for Future. Unlike the methods on the Future class itself, the pure and flatMap methods on the monad can’t accept implicit ExecutionContext parameters (because the parameters aren’t part of the definitions in the Monad trait). To work around this, Cats requires us to have an ExecutionContext in scope when we summon a Monad for Future:

It’s difficult to demonstrate the flatMap and map methods directly on Scala monads like Option and List, because they define their own explicit versions of those methods. Instead we’ll write a generic function that performs a calculation on parameters that come wrapped in a monad of the user’s choice:

We can rewrite this code using for comprehensions. The compiler will “do the right thing” by rewriting our comprehension in terms of flatMap and map and inserting the correct implicit conversions to use our Monad:

It would be incredibly useful if we could use sumSquare with parameters that were either in a monad or not in a monad at all. This would allow us to abstract over monadic and non-monadic code. Fortunately, Cats provides the Id type to bridge the gap:

Id allows us to call our monadic method using plain values. However, the exact semantics are difficult to understand. We cast the parameters to sumSquare as Id[Int] and received an Id[Int] back as a result!

What’s going on? Here is the definition of Id to explain:

package cats
type Id[A] = A

Id is actually a type alias that turns an atomic type into a single-parameter type constructor. We can cast any value of any type to a corresponding Id:

The ability to abstract over monadic and non-monadic code is extremely powerful. For example, we can run code asynchronously in production using Future and synchronously in test using Id. We’ll see this in our first case study in Chapter 8.

4.3.1 Exercise: Monadic Secret Identities

Implement pure, map, and flatMap for Id! What interesting discoveries do you uncover about the implementation?

The map method takes a parameter of type Id[A], applies a function of type A => B, and returns an Id[B]. But Id[A] is simply A and Id[B] is simply B! All we have to do is call the function—no packing or unpacking required:

This ties in with our understanding of functors and monads as sequencing type classes. Each type class allows us to sequence operations ignoring some kind of complication. In the case of Id there is no complication, making map and flatMap the same thing.

Notice that we haven’t had to write type annotations in the method bodies above. The compiler is able to interpret values of type A as Id[A] and vice versa by the context in which they are used.

The only restriction we’ve seen to this is that Scala cannot unify types and type constructors when searching for implicits. Hence our need to re-type Int as Id[Int] in the call to sumSquare at the opening of this section:

sumSquare(3 : Id[Int], 4 : Id[Int])

4.4 Either

Let’s look at another useful monad: the Either type from the Scala standard library. In Scala 2.11 and earlier, many people didn’t consider Either a monad because it didn’t have map and flatMap methods. In Scala 2.12, however, Either became right biased.

4.4.1 Left and Right Bias

In Scala 2.11, Either had no default map or flatMap method. This made the Scala 2.11 version of Either inconvenient to use in for comprehensions. We had to insert calls to .right in every generator clause:

In Scala 2.12, Either was redesigned. The modern Either makes the decision that the right side represents the success case and thus supports map and flatMap directly. This makes for comprehensions much more pleasant:

Cats back-ports this behaviour to Scala 2.11 via the cats.syntax.either import, allowing us to use right-biased Either in all supported versions of Scala. In Scala 2.12+ we can either omit this import or leave it in place without breaking anything:

These “smart constructors” have advantages over Left.apply and Right.apply because they return results of type Either instead of Left and Right. This helps avoid type inference bugs caused by over-narrowing, like the bug in the example below:

This approach solves the problems we saw with Throwable. It gives us a fixed set of expected error types and a catch-all for anything else that we didn’t expect. We also get the safety of exhaustivity checking on any pattern matching we do:

4.4.5 Exercise: What is Best?

Is the error handling strategy in the previous examples well suited for all purposes? What other features might we want from error handling?

This is an open question. It’s also kind of a trick question—the answer depends on the semantics we’re looking for. Some points to ponder:

Error recovery is important when processing large jobs. We don’t want to run a job for a day and then find it failed on the last element.

Error reporting is equally important. We need to know what went wrong, not just that something went wrong.

In a number of cases, we want to collect all the errors, not just the first one we encountered. A typical example is validating a web form. It’s a far better experience to report all errors to the user when they submit a form than to report them one at a time.

4.5 Aside: Error Handling and MonadError

Cats provides an additional type class called MonadError that abstracts over Either-like data types that are used for error handling. MonadError provides extra operations for raising and handling errors.

This Section is Optional!

You won’t need to use MonadError unless you need to abstract over error handling monads. For example, you can use MonadError to abstract over Future and Try, or over Either and EitherT (which we will meet in Chapter 5).

If you don’t need this kind of abstraction right now, feel free to skip onwards to Section 4.6.

In reality, MonadError extends another type class called ApplicativeError. However, we won’t encounter Applicatives until Chapter 6. The semantics are the same for each type class so we can ignore this detail for now.

4.5.2 Raising and Handling Errors

The two most important methods of MonadError are raiseError and handleError. raiseError is like the pure method for Monad except that it creates an instance representing a failure:

There is also a third useful method called ensure that implements filter-like behaviour. We test the value of a successful monad with a predicate and specify an error to raise if the predicate returns false:

4.5.3 Instances of MonadError

Cats provides instances of MonadError for numerous data types including Either, Future, and Try. The instance for Either is customisable to any error type, whereas the instances for Future and Try always represent errors as Throwables:

4.5.4 Exercise: Abstracting

4.6 The Eval Monad

cats.Eval is a monad that allows us to abstract over different models of evaluation. We typically hear of two such models: eager and lazy. Eval throws in a further distinction of whether or not a result is memoized.

4.6.1 Eager, Lazy, Memoized, Oh My!

What do these terms mean?

Eager computations happen immediately whereas lazy computations happen on access. Memoized computations are run once on first access, after which the results are cached.

For example, Scala vals are eager and memoized. We can see this using a computation with a visible side-effect. In the following example, the code to compute the value of x happens at the definition site rather than on access (eager). Accessing x recalls the stored value without re-running the code (memoized).

Last but not least, lazy vals are lazy and memoized. The code to compute z below is not run until we access it for the first time (lazy). The result is then cached and re-used on subsequent accesses (memoized):

4.6.3 Eval as a Monad

Like all monads, Eval'smap and flatMap methods add computations to a chain. In this case, however, the chain is stored explicitly as a list of functions. The functions aren’t run until we call Eval'svalue method to request a result:

Eval has a memoize method that allows us to memoize a chain of computations. The result of the chain up to the call to memoize is cached, whereas calculations after the call retain their original semantics:

4.6.4 Trampolining and Eval.defer

One useful property of Eval is that its map and flatMap methods are trampolined. This means we can nest calls to map and flatMap arbitrarily without consuming stack frames. We call this property “stack safety”.

Oops! That didn’t work—our stack still blew up! This is because we’re still making all the recursive calls to factorial before we start working with Eval'smap method. We can work around this using Eval.defer, which takes an existing instance of Eval and defers its evaluation. The defer method is trampolined like map and flatMap, so we can use it as a quick way to make an existing operation stack safe:

Eval is a useful tool to enforce stack safety when working on very large computations and data structures. However, we must bear in mind that trampolining is not free. It avoids consuming stack by creating a chain of function objects on the heap. There are still limits on how deeply we can nest computations, but they are bounded by the size of the heap rather than the stack.

4.6.5 Exercise: Safer Folding using Eval

The naive implementation of foldRight below is not stack safe. Make it so using Eval:

The easiest way to fix this is to introduce a helper method called foldRightEval. This is essentially our original method with every occurrence of B replaced with Eval[B], and a call to Eval.defer to protect the recursive call:

4.7 The Writer Monad

cats.data.Writer is a monad that lets us carry a log along with a computation. We can use it to record messages, errors, or additional data about a computation, and extract the log alongside the final result.

One common use for Writers is recording sequences of steps in multi-threaded computations where standard imperative logging techniques can result in interleaved messages from different contexts. With Writer the log for the computation is tied to the result, so we can run concurrent computations without mixing logs.

Cats Data Types

Writer is the first data type we’ve seen from the cats.data package. This package provides instances of various type classes that produce useful semantics. Other examples from cats.data include the monad transformers that we will see in the next chapter, and the Validated type we will encounter in Chapter 6.

4.7.1 Creating and Unpacking Writers

A Writer[W, A] carries two values: a log of type W and a result of type A. We can create a Writer from values of each type as follows:

import cats.data.Writerimport cats.instances.vector._ // for Monoid
Writer(Vector(
"It was the best of times",
"it was the worst of times"
), 1859)
// res0: cats.data.WriterT[cats.Id,scala.collection.immutable.Vector[String],Int] = WriterT((Vector(It was the best of times, it was the worst of times),1859))

Notice that the type reported on the console is actually WriterT[Id, Vector[String], Int] instead of Writer[Vector[String], Int] as we might expect. In the spirit of code reuse, Cats implements Writer in terms of another type, WriterT. WriterT is an example of a new concept called a monad transformer, which we will cover in the next chapter.

Let’s try to ignore this detail for now. Writer is a type alias for WriterT, so we can read types like WriterT[Id, W, A] as Writer[W, A]:

type Writer[W, A] = WriterT[Id, W, A]

For convenience, Cats provides a way of creating Writers specifying only the log or the result. If we only have a result we can use the standard pure syntax. To do this we must have a Monoid[W] in scope so Cats knows how to produce an empty log:

4.7.2 Composing and Transforming Writers

The log in a Writer is preserved when we map or flatMap over it. flatMap appends the logs from the source Writer and the result of the user’s sequencing function. For this reason it’s good practice to use a log type that has an efficient append and concatenate operations, such as a Vector:

We can transform both log and result simultaneously using bimap or mapBoth. bimap takes two function parameters, one for the log and one for the result. mapBoth takes a single function that accepts two parameters:

4.8 The Reader Monad

cats.data.Reader is a monad that allows us to sequence operations that depend on some input. Instances of Reader wrap up functions of one argument, providing us with useful methods for composing them.

One common use for Readers is dependency injection. If we have a number of operations that all depend on some external configuration, we can chain them together using a Reader to produce one large operation that accepts the configuration as a parameter and runs our program in the order specified.

4.8.1 Creating and Unpacking Readers

We can create a Reader[A, B] from a function A => B using the Reader.apply constructor:

So far so simple, but what advantage do Readers give us over the raw functions?

4.8.2 Composing Readers

The power of Readers comes from their map and flatMap methods, which represent different kinds of function composition. We typically create a set of Readers that accept the same type of configuration, combine them with map and flatMap, and then call run to inject the config at the end.

The map method simply extends the computation in the Reader by passing its result through a function:

4.8.3 Exercise: Hacking on Readers

The classic use of Readers is to build programs that accept a configuration as a parameter. Let’s ground this with a complete example of a simple login system. Our configuration will consist of two databases: a list of valid users and a list of their passwords:

Remember: the idea is to leave injecting the configuration until last. This means setting up functions that accept the config as a parameter and check it against the concrete user info we have been given:

4.8.4 When to Use Readers?

Readers provide a tool for doing dependency injection. We write steps of our program as instances of Reader, chain them together with map and flatMap, and build a function that accepts the dependency as input.

There are many ways of implementing dependency injection in Scala, from simple techniques like methods with multiple parameter lists, through implicit parameters and type classes, to complex techniques like the cake pattern and DI frameworks.

Readers are most useful in situations where:

we are constructing a batch program that can easily be represented by a function;

we need to defer injection of a known parameter or set of parameters;

we want to be able to test parts of the program in isolation.

By representing the steps of our program as Readers we can test them as easily as pure functions, plus we gain access to the map and flatMap combinators.

For more advanced problems where we have lots of dependencies, or where a program isn’t easily represented as a pure function, other dependency injection techniques tend to be more appropriate.

Kleisli Arrows

You may have noticed from console output that Reader is implemented in terms of another type called Kleisli. Kleisli arrows provide a more general form of Reader that generalise over the type constructor of the result type. We will encounter Kleislis again in Chapter 5.

4.9 The State Monad

cats.data.State allows us to pass additional state around as part of a computation. We define State instances representing atomic state operations and thread them together using map and flatMap. In this way we can model mutable state in a purely functional way, without using mutation.

4.9.1 Creating and Unpacking State

Boiled down to their simplest form, instances of State[S, A] represent functions of type S => (S, A). S is the type of the state and A is the type of the result.

In other words, an instance of State is a function that does two things:

transforms an input state to an output state;

computes a result.

We can “run” our monad by supplying an initial state. State provides three methods—run, runS, and runA—that return different combinations of state and result. Each method returns an instance of Eval, which State uses to maintain stack safety. We call the value method as usual to extract the actual result:

// Get the state and the result:val (state, result) = a.run(10).value// state: Int = 10// result: String = The state is 10// Get the state, ignore the result:val state = a.runS(10).value// state: Int = 10// Get the result, ignore the state:val result = a.runA(10).value// result: String = The state is 10

4.9.2 Composing and Transforming State

As we’ve seen with Reader and Writer, the power of the State monad comes from combining instances. The map and flatMap methods thread the state from one instance to another. Each individual instance represents an atomic state transformation, and their combination represents a complete sequence of changes:

As you can see, in this example the final state is the result of applying both transformations in sequence. State is threaded from step to step even though we don’t interact with it in the for comprehension.

The general model for using the State monad is to represent each step of a computation as an instance and compose the steps using the standard monad operators. Cats provides several convenience constructors for creating primitive steps:

4.9.3 Exercise: Post-Order Calculator

The State monad allows us to implement simple interpreters for complex expressions, passing the values of mutable registers along with the result. We can see a simple example of this by implementing a calculator for post-order integer arithmetic expressions.

In case you haven’t heard of post-order expressions before (don’t worry if you haven’t), they are a mathematical notation where we write the operator after its operands. So, for example, instead of writing 1 + 2 we would write:

12 +

Although post-order expressions are difficult for humans to read, they are easy to evaluate in code. All we need to do is traverse the symbols from left to right, carrying a stack of operands with us as we go:

when we see a number, we push it onto the stack;

when we see an operator, we pop two operands off the stack, operate on them, and push the result in their place.

This allows us to evaluate complex expressions without using parentheses. For example, we can evaluate (1 + 2) * 3) as follows:

Let’s write an interpreter for these expressions. We can parse each symbol into a State instance representing a transformation on the stack and an intermediate result. The State instances can be threaded together using flatMap to produce an interpreter for any sequence of symbols.

Start by writing a function evalOne that parses a single symbol into an instance of State. Use the code below as a template. Don’t worry about error handling for now—if the stack is in the wrong configuration, it’s OK to throw an exception.

If this seems difficult, think about the basic form of the State instances you’re returning. Each instance represents a functional transformation from a stack to a pair of a stack and a result. You can ignore any wider context and focus on just that one step:

The operator function is a little more complex. We have to pop two operands off the stack (having the second operand at the top of the stack) and push the result in their place. The code can fail if the stack doesn’t have enough operands on it, but the exercise description allows us to throw an exception in this case:

evalOne allows us to evaluate single-symbol expressions as follows. We call runA supplying Nil as an initial stack, and call value to unpack the resulting Eval instance:

evalOne("42").runA(Nil).value// res3: Int = 42

We can represent more complex programs using evalOne, map, and flatMap. Note that most of the work is happening on the stack, so we ignore the results of the intermediate steps for evalOne("1") and evalOne("2"):

Generalise this example by writing an evalAll method that computes the result of a List[String]. Use evalOne to process each symbol, and thread the resulting State monads together using flatMap. Your function should have the following signature:

defevalAll(input: List[String]): CalcState[Int] =
???

We implement evalAll by folding over the input. We start with a pure CalcState that returns 0 if the list is empty. We flatMap at each stage, ignoring the intermediate results as we saw in the example:

Because evalOne and evalAll both return instances of State, we can thread these results together using flatMap. evalOne produces a simple stack transformation and evalAll produces a complex one, but they’re both pure functions and we can use them in any order as many times as we like:

4.10 Defining Custom Monads

We can define a Monad for a custom type by providing implementations of three methods: flatMap, pure, and a method we haven’t seen yet called tailRecM. Here is an implementation of Monad for Option as an example:

The tailRecM method is an optimisation used in Cats to limit the amount of stack space consumed by nested calls to flatMap. The technique comes from a 2015 paper by PureScript creator Phil Freeman. The method should recursively call itself until the result of fn returns a Right.

If we can make tailRecM tail-recursive, Cats is able to guarantee stack safety in recursive situations such as folding over large lists (see Section 7.1). If we can’t make tailRecM tail-recursive, Cats cannot make these guarantees and extreme use cases may result in StackOverflowErrors. All of the built-in monads in Cats have tail-recursive implementations of tailRecM, although writing one for custom monads can be a challenge… as we shall see.

4.10.1 Exercise: Branching out Further with Monads

Let’s write a Monad for our Tree data type from last chapter. Here’s the type again:

The solution above is perfectly fine for this exercise. Its only downside is that Cats cannot make guarantees about stack safety.

The tail-recursive solution is much harder to write. We adapted this solution from this Stack Overflow post by Nazarii Bardiuk. It involves an explicit depth first traversal of the tree, maintaining an open list of nodes to visit and a closed list of nodes to use to reconstruct the tree:

The monad for Option provides fail-fast semantics. The monad for List provides concatenation semantics. What are the semantics of flatMap for a binary tree? Every node in the tree has the potential to be replaced with a whole subtree, producing a kind of “growing” or “feathering” behaviour, reminiscent of list concatenation along two axes.

4.11 Summary

In this chapter we’ve seen monads up-close. We saw that flatMap can be viewed as an operator for sequencing computations, dictating the order in which operations must happen. From this viewpoint, Option represents a computation that can fail without an error message, Either represents computations that can fail with a message, List represents multiple possible results, and Future represents a computation that may produce a value at some point in the future.

We’ve also seen some of the custom types and data structures that Cats provides, including Id, Reader, Writer, and State. These cover a wide range of use cases.

Finally, in the unlikely event that we have to implement a custom monad, we’ve learned about defining our own instance using tailRecM. tailRecM is an odd wrinkle that is a concession to building a functional programming library that is stack-safe by default. We don’t need to understand tailRecM to understand monads, but having it around gives us benefits of which we can be grateful when writing monadic code.

5 Monad Transformers

Monads are like burritos, which means that once you acquire a taste, you’ll find yourself returning to them again and again. This is not without issues. As burritos can bloat the waist, monads can bloat the code base through nested for-comprehensions.

Imagine we are interacting with a database. We want to look up a user record. The user may or may not be present, so we return an Option[User]. Our communication with the database could fail for many reasons (network issues, authentication problems, and so on), so this result is wrapped up in an Either, giving us a final result of Either[Error, Option[User]].

To use this value we must nest flatMap calls (or equivalently, for-comprehensions):

It is impossible to write a general definition of flatMap without knowing something about M1 or M2. However, if we do know something about one or other monad, we can typically complete this code. For example, if we fix M2 above to be Option, a definition of flatMap comes to light:

Notice that the definition above makes use of None—an Option-specific concept that doesn’t appear in the general Monad interface. We need this extra detail to combine Option with other monads. Similarly, there are things about other monads that help us write composed flatMap methods for them. This is the idea behind monad transformers: Cats defines transformers for a variety of monads, each providing the extra knowledge we need to compose that monad with others. Let’s look at some examples.

5.2 A Transformative Example

Cats provides transformers for many monads, each named with a T suffix: EitherT composes Either with other monads, OptionT composes Option, and so on.

Here’s an example that uses OptionT to compose List and Option. We can use OptionT[List, A], aliased to ListOption[A] for convenience, to transform a List[Option[A]] into a single monad:

import cats.data.OptionTtype ListOption[A] = OptionT[List, A]

Note how we build ListOption from the inside out: we pass List, the type of the outer monad, as a parameter to OptionT, the transformer for the inner monad.

We can create instances of ListOption using the OptionT constructor, or more conveniently using pure:

This is the basis of all monad transformers. The combined map and flatMap methods allow us to use both component monads without having to recursively unpack and repack values at each stage in the computation. Now let’s look at the API in more depth.

Complexity of Imports

The imports in the code samples above hint at how everything bolts together.

We import cats.syntax.applicative to get the pure syntax. pure requires an implicit parameter of type Applicative[ListOption]. We haven’t met Applicatives yet, but all Monads are also Applicatives so we can ignore that difference for now.

In order to generate our Applicative[ListOption] we need instances of Applicative for List and OptionT. OptionT is a Cats data type so its instance is provided by its companion object. The instance for List comes from cats.instances.list.

Notice we’re not importing cats.syntax.functor or cats.syntax.flatMap. This is because OptionT is a concrete data type with its own explicit map and flatMap methods. It wouldn’t cause problems if we imported the syntax—the compiler would ignore it in favour of the explicit methods.

Remember that we’re subjecting ourselves to these shenanigans because we’re stubbornly refusing to use the universal Cats import, cats.implicits. If we did use that import, all of the instances and syntax we needed would be in scope and everything would just work.

5.3 Monad Transformers in Cats

Each monad transformer is a data type, defined in cats.data, that allows us to wrap stacks of monads to produce new monads. We use the monads we’ve built via the Monad type class. The main concepts we have to cover to understand monad transformers are:

the available transformer classes;

how to build stacks of monads using transformers;

how to construct instances of a monad stack; and

how to pull apart a stack to access the wrapped monads.

5.3.1 The Monad Transformer Classes

By convention, in Cats a monad Foo will have a transformer class called FooT. In fact, many monads in Cats are defined by combining a monad transformer with the Id monad. Concretely, some of the available instances are:

In Section 4.8 we mentioned that the Reader monad was a specialisation of a more general concept called a “kleisli arrow”, represented in Cats as cats.data.Kleisli.

We can now reveal that Kleisli and ReaderT are, in fact, the same thing! ReaderT is actually a type alias for Kleisli. Hence, we were creating Readers last chapter and seeing Kleislis on the console.

5.3.2 Building Monad Stacks

All of these monad transformers follow the same convention. The transformer itself represents the inner monad in a stack, while the first type parameter specifies the outer monad. The remaining type parameters are the types we’ve used to form the corresponding monads.

For example, our ListOption type above is an alias for OptionT[List, A] but the result is effectively a List[Option[A]]. In other words, we build monad stacks from the inside out:

type ListOption[A] = OptionT[List, A]

Many monads and all transformers have at least two type parameters, so we often have to define type aliases for intermediate stages.

For example, suppose we want to wrap Either around Option. Option is the innermost type so we want to use the OptionT monad transformer. We need to use Either as the first type parameter. However, Either itself has two type parameters and monads only have one. We need a type alias to convert the type constructor to the correct shape:

Things become even more confusing when we want to stack three or more monads.

For example, let’s create a Future of an Either of Option. Once again we build this from the inside out with an OptionT of an EitherT of Future. However, we can’t define this in one line because EitherT has three type parameters:

caseclass EitherT[F[_], E, A](stack: F[Either[E, A]]) {
// etc...
}

The three type parameters are as follows:

F[_] is the outer monad in the stack (Either is the inner);

E is the error type for the Either;

A is the result type for the Either.

This time we create an alias for EitherT that fixes Future and Error and allows A to vary:

Each call to value unpacks a single monad transformer. We may need more than one call to completely unpack a large stack. For example, to Await the FutureEitherOption stack above, we need to call value twice:

5.3.4 Default Instances

Many monads in Cats are defined using the corresponding transformer and the Id monad. This is reassuring as it confirms that the APIs for monads and transformers are identical. Reader, Writer, and State are all defined in this way:

In other cases monad transformers are defined separately to their corresponding monads. In these cases, the methods of the transformer tend to mirror the methods on the monad. For example, OptionT defines getOrElse, and EitherT defines fold, bimap, swap, and other useful methods.

5.3.5 Usage Patterns

Widespread use of monad transformers is sometimes difficult because they fuse monads together in predefined ways. Without careful thought, we can end up having to unpack and repack monads in different configurations to operate on them in different contexts.

We can cope with this in multiple ways. One approach involves creating a single “super stack” and sticking to it throughout our code base. This works if the code is simple and largely uniform in nature. For example, in a web application, we could decide that all request handlers are asynchronous and all can fail with the same set of HTTP error codes. We could design a custom ADT representing the errors and use a fusion Future and Either everywhere in our code:

The “super stack” approach starts to fail in larger, more heterogeneous code bases where different stacks make sense in different contexts. Another design pattern that makes more sense in these contexts uses monad transformers as local “glue code”. We expose untransformed stacks at module boundaries, transform them to operate on them locally, and untransform them before passing them on. This allows each module of code to make its own decisions about which transformers to use:

Unfortunately, there aren’t one-size-fits-all approaches to working with monad transformers. The best approach for you may depend on a lot of factors: the size and experience of your team, the complexity of your code base, and so on. You may need to experiment and gather feedback from colleagues to determine whether monad transformers are a good fit.

5.4 Exercise: Monads: Transform and Roll Out

The Autobots, well-known robots in disguise, frequently send messages during battle requesting the power levels of their team mates. This helps them coordinate strategies and launch devastating attacks. The message sending method looks like this:

defgetPowerLevel(autobot: String): Response[Int] =
???

Transmissions take time in Earth’s viscous atmosphere, and messages are occasionally lost due to satellite malfunction or sabotage by pesky Decepticons8. Responses are therefore represented as a stack of monads:

Two autobots can perform a special move if their combined power level is greater than 15. Write a second method, canSpecialMove, that accepts the names of two allies and checks whether a special move is possible. If either ally is unavailable, fail with an appropriate error message:

5.5 Summary

In this chapter we introduced monad transformers, which eliminate the need for nested for comprehensions and pattern matching when working with “stacks” of nested monads.

Each monad transformer, such as FutureT, OptionT or EitherT, provides the code needed to merge its related monad with other monads. The transformer is a data structure that wraps a monad stack, equipping it with map and flatMap methods that unpack and repack the whole stack.

The type signatures of monad transformers are written from the inside out, so an EitherT[Option, String, A] is a wrapper for an Option[Either[String, A]]. It is often useful to use type aliases when writing transformer types for deeply nested monads.

With this look at monad transformers, we have now covered everything we need to know about monads and the sequencing of computations using flatMap. In the next chapter we will switch tack and discuss two new type classes, Semigroupal and Applicative, that support new kinds of operation such as zipping independent values within a context.

6 Semigroupal and Applicative

In previous chapters we saw how functors and monads let us sequence operations using map and flatMap. While functors and monads are both immensely useful abstractions, there are certain types of program flow that they cannot represent.

One such example is form validation. When we validate a form we want to return all the errors to the user, not stop on the first error we encounter. If we model this with a monad like Either, we fail fast and lose errors. For example, the code below fails on the first call to parseInt and doesn’t go any further:

Another example is the concurrent evaluation of Futures. If we have several long-running independent tasks, it makes sense to execute them concurrently. However, monadic comprehension only allows us to run them in sequence. map and flatMap aren’t quite capable of capturing what we want because they make the assumption that each computation is dependent on the previous one:

The calls to parseInt and Future.apply above are independent of one another, but map and flatMap can’t exploit this. We need a weaker construct—one that doesn’t guarantee sequencing—to achieve the result we want. In this chapter we will look at two type classes that support this pattern:

Semigroupal encompasses the notion of composing pairs of contexts. Cats provides a cats.syntax.apply module that makes use of Semigroupal and Functor to allow users to sequence functions with multiple arguments.

Applicative extends Semigroupal and Functor. It provides a way of applying functions to parameters within a context. Applicative is the source of the pure method we introduced in Chapter 4.

Applicatives are often formulated in terms of function application, instead of the semigroupal formulation that is emphasised in Cats. This alternative formulation provides a link to other libraries and languages such as Scalaz and Haskell. We’ll take a look at different formulations of Applicative, as well as the relationships between Semigroupal, Functor, Applicative, and Monad, towards the end of the chapter.

6.1 Semigroupal

cats.Semigroupal is a type class that allows us to combine contexts9. If we have two objects of type F[A] and F[B], a Semigroupal[F] allows us to combine them to form an F[(A, B)]. Its definition in Cats is:

As we discussed at the beginning of this chapter, the parameters fa and fb are independent of one another: we can compute them in either order before passing them to product. This is in contrast to flatMap, which imposes a strict order on its parameters. This gives us more freedom when defining instances of Semigroupal than we get when defining Monads.

6.1.1 Joining Two Contexts

While Semigroup allows us to join values, Semigroupal allows us to join contexts. Let’s join some Options as an example:

6.3 Semigroupal Applied to Different Types

Semigroupal doesn’t always provide the behaviour we expect, particularly for types that also have instances of Monad. We have seen the behaviour of the Semigroupal for Option. Let’s look at some examples for other types.

Future

The semantics for Future provide parallel as opposed to sequential execution:

This is perhaps surprising. Zipping lists tends to be a more common operation. We’ll see why we get this behaviour in a moment.

Either

We opened this chapter with a discussion of fail-fast versus accumulating error-handling. We might expect product applied to Either to accumulate errors instead of fail fast. Again, perhaps surprisingly, we find that product implements the same fail-fast behaviour as flatMap:

In this example product sees the first failure and stops, even though it is possible to examine the second parameter and see that it is also a failure.

6.3.1 Semigroupal Applied to Monads

The reason for the surprising results for List and Either is that they are both monads. To ensure consistent semantics, Cats’ Monad (which extends Semigroupal) provides a standard definition of product in terms of map and flatMap. This gives what we might think of as unexpected and less useful behaviour for a number of data types. The consistency of semantics is important for higher level abstractions, but we don’t know about those yet.

Even our results for Future are a trick of the light. flatMap provides sequential ordering, so product provides the same. The parallel execution we observe occurs because our constituent Futures start running before we call product. This is equivalent to the classic create-then-flatMap pattern:

So why bother with Semigroupal at all? The answer is that we can create useful data types that have instances of Semigroupal (and Applicative) but not Monad. This frees us to implement product in different ways. We’ll examine this further in a moment when we look at an alternative data type for error handling.

6.4 Validated

By now we are familiar with the fail-fast error handling behaviour of Either. Furthermore, because Either is a monad, we know that the semantics of product are the same as those for flatMap. In fact, it is impossible for us to design a monadic data type that implements error accumulating semantics without breaking the consistency of these two methods.

Fortunately, Cats provides a data type called Validated that has an instance of Semigroupal but no instance of Monad. The implementation of product is therefore free to accumulate errors:

Validated complements Either nicely. Between the two we have support for both of the common types of error handling: fail-fast and accumulating.

6.4.1 Creating Instances of Validated

Validated has two subtypes, Validated.Valid and Validated.Invalid, that correspond loosely to Right and Left. There are a lot of ways to create instances of these types. We can create them directly using their apply methods:

6.4.2 Combining Instances of Validated

We can combine instances of Validated using any of the methods or syntax described for Semigroupal above.

All of these techniques require an instance of Semigroupal to be in scope. As with Either, we need to fix the error type to create a type constructor with the correct number of parameters for Semigroupal:

type AllErrorsOr[A] = Validated[String, A]

Validated accumulates errors using a Semigroup, so we need one of those in scope to summon the Semigroupal. If no Semigroup is visible at the call site, we get an annoyingly unhelpful compilation error:

6.4.3 Methods of Validated

Validated comes with a suite of methods that closely resemble those available for Either, including the methods from cats.syntax.either. We can use map, leftMap, and bimap to transform the values inside the valid and invalid sides:

We can’t flatMap because Validated isn’t a monad. However, Cats does provide a stand-in for flatMap called andThen. The type signature of andThen is identical to that of flatMap, but it has a different name because it is not a lawful implementation with respect to the monad laws:

6.4.4 Exercise: Form Validation

Let’s get used to Validated by implementing a simple HTML registration form. We receive request data from the client in a Map[String, String] and we want to parse it to create a User object:

caseclassUser(name: String, age: Int)

Our goal is to implement code that parses the incoming data enforcing the following rules:

the name and age must be specified;

the name must not be blank;

the age must be a valid non-negative integer.

If all the rules pass our parser we should return a User. If any rules fail we should return a List of the error messages.

To implement this example we’ll need to combine rules in sequence and in parallel. We’ll use Either to combine computations in sequence using fail-fast semantics, and Validated to combine them in parallel using accumulating semantics.

Let’s start with some sequential combination. We’ll define two methods to read the "name" and "age" fields:

readName will take a Map[String, String] parameter, extract the "name" field, check the relevant validation rules, and return an Either[List[String], String].

readAge will take a Map[String, String] parameter, extract the "age" field, check the relevant validation rules, and return an Either[List[String], Int].

We’ll build these methods up from smaller building blocks. Start by defining a method getValue that reads a String from the Map given a field name.

The need to switch back and forth between Either and Validated is annoying. The choice of whether to use Either or Validated as a default is determined by context. In application code, we typically find areas that favour accumulating semantics and areas that favour fail-fast semantics. We pick the data type that best suits our need and switch to the other as necessary in specific situations.

6.5 Apply and Applicative

Semigroupals aren’t mentioned frequently in the wider functional programming literature. They provide a subset of the functionality of a related type class called an applicative functor (“applicative” for short).

Semigroupal and Applicative effectively provide alternative encodings of the same notion of joining contexts. Both encodings are introduced in the same 2008 paper by Conor McBride and Ross Paterson10.

Cats models applicatives using two type classes. The first, cats.Apply, extends Semigroupal and Functor and adds an ap method that applies a parameter to a function within a context. The second, cats.Applicative, extends Apply, adds the pure method introduced in Chapter 4. Here’s a simplified definition in code: