#17: Benchmarking with Benchee

Elixir is fast, but how can we distinguish between the performance of two pieces of code that have the same output?

Benchmarking is the answer and Benchee is a great framework for benchmarking in Elixir.

In this episode we’ll use Benchee to benchmark some code and see what’s faster.

Let’s get started.

We’ll open our mixfile and add ‘benchee’ as a dependency. And we’ll set the only: :dev option since we only want it in our development environment.

mix.exs
defp deps do
[{:benchee, "~> 0.9", only: :dev}]
end

Then we’ll go to the command line and download our dependencies.

$ mix deps.get

Now we need something to benchmark.

Let’s write a recursive function that takes a list of numbers, adds one to it, and then returns the updated list. We’ll compare it against Enum.map/2.

First let’s define a new function we’ll call ‘incrementer’.

It will take a list as its first parameter. We’ll match against it to get the ‘head’, which will be the first number in the list, and the ‘tail’, which will be the remainder of the list.

The second parameter will be an accumulator, which will contain our updated list.

Since this is a recursive function our ‘incremental’ function will then call itself. We’ll pass in the tail of our list and then build our updated list by adding one to the ‘head’ of our list and then adding it to the accumulator.

And since a recursive call is the last expression, our function is tail call optimized.

Now we’ll need to create another function that pattern matches against an empty list. This will happen when we’ve processed all the items in our list.

And in this function we’ll then call Enum.reverse/1 since we want the order of the list to match.

And finally let’s add a function head that will let us set a default value for the accumulator, which we’ll set as an empty list.

We’ll create a directory named ‘benchmarks’ and in it a file named ‘increment_example.exs’

First let’s create our list of numbers. Let’s create one with numbers ‘1’ through ‘100,000’.

Now we’ll call Benchee.run/2 .

We’ll pass in a map of the functions we want to benchmark. With the key being a readable name and the value a function with the code we want to benchmark.

Let’s first include our ‘tail call optimized’ function and then we’ll define a function that calls our Teacher.incrementer/2 function with the list defined above.

The second function we want to benchmark is Enum.map/2 so let’s give it a name and then we’ll define another function that calls Enum.map/2 and again uses the list we defined above, we’ll then give it a function that simply takes the number from the list and increments it.

And we can see our benchmark ran with our updated options - parallel set to 2 and a time of 10 seconds.

Now that we know how to customize our benchmark, let’s remove our customizations and just use the defaults for now.

A nice bonus of using Benchee is there are a few different plugins we can use to provide nice visualizations of the benchmarks we run. One of them is ‘benchee_html’, which will take our benchmarks and display them in graphs saved in a standalone HTML files.

Let’s add benchee_html to our project.

We’ll open the ‘mixfile’ and add benchee_html as a dependency, with the only: :dev option since we only want it in our development environment.

It prints that our results were generated, but the console results are missing. That’s because they’re included by default and now that we’re explicitly defining our formatters we’ll have to include it as well.