How to write RSpec formatters from scratch

Posted by Ilija Eftimov on April 28, 2015

Recently I did an experiment with RSpec’s formatters. Turns out, the output that RSpec returns when you run your specs can be very customized for your own needs. Read on to learn how you can write custom RSpec formatters.

Writing custom formatters

RSpec allows customization of the output by creating your own Formatter class. Yep, it’s that easy. You just need to write one class and than require it into RSpec’s configuration to use it. Lets explore couple of key concepts about the formatters and it’s internals.

The Protocol

First thing to be aware of is the protocol (the order) that RSpec uses when calling methods from the formatter. Keep in mind that every method receives one argument, which is a Notification object. This is sent by the RSpec reporter which notifies the formatter of the outcome of the example. On the beginning Formatter#start is called. This method takes a notification argument of the class StartNotification. On every example group, Formatter#example_group_started is called. This method takes a notification argument of the class GroupNotification. When an example is ran, one of these methods is called, based on the result of the example:

If the example passes, Formatter#example_passed is called. The notification argument is of class ExampleNotification.

So, say we want a formatter that shows the progress of the suite, just like the built-in progress formatter, but it will group the failing and the pending specs in the summary of the suite. Oh well, a picture is worth a thousand words:

The picture shows what we’ll be working on in the rest of this post. We’ll call this formatter GroupingFormatter.

As you can see, the GroupingFormatter is just a class. In it’s initializer it takes the output as an argument and sets it as an instance variable. Also, on line 2, you can see the aforementioned RSpec.register call. We pass self as the first argument, because we want to register this class as a formatter. The rest of the arguments are method names that RSpec will call when using this formatter. What this means is that when you define a method for the protocol, if you don’t register it - it will not be called. Basically, RSpec won’t know it exists at all. Next, the dump_summary method calls the duration method on the notification object, which returns a number representing the time of the specs’ duration in seconds. So, how can we test if this is working? The command is:

In the dump_summary method we use the RSpec::Core::Formatters::Helpers module which has some methods that can help us turn the duration number into a meaningful string. The output now looks like:

Finished in 0.00758 seconds.

Okay, great. Now, lets make this formatter mimic the reporting formatter that comes with RSpec. We need the formatter to show a dot for every passing example, F for every failing example and an asterisk for every pending example.

So, the reporter (the algorithm that follows the protocol) will call example_failed when an example fails, example_pending when an example is pending and example_passed when an example passes. This is really self-explanatory - we add the case specific character to the output for every example. Take note that I added the method names to the RSpec.register call. If I didn’t - they’d be ignored. The output will now look like:

.....FF*..
Finished in 0.0207 seconds.

Looking good, things are starting to take shape! Now for the more complicated part. How can we group the pending/failing specs? First, lets group the pending specs.

Lets look at the dump_pending method now. First, it adds “PENDING” to the output. Next, it loops through the pending_examples array and creates an array of strings for each of the pending examples. Note that I added the new method to the RSpec.register call, it would be ignored otherwise. Each string in the array will look something like this:

Something is pending - ./something_spec.rb:90

At the end, we call _join _on the array of strings to build a single formatted string that we append to the output. Now when we run the specs with the formatter, the output will look like:

Looking good. Now, for the trickiest part, grouping the failing specs and adding the error underneath every failing spec.

classGroupingFormatterRSpec::Core::Formatters.registerself,:dump_pending,:dump_failures,:close,:dump_summary,:example_passed,:example_failed,:example_pendingdefinitializeoutput@output=outputenddefexample_passednotification# ExampleNotification@output<<"."enddefexample_failednotification# FailedExampleNotification@output<<"F"enddefexample_pendingnotification# ExampleNotification@output<<"*"enddefdump_pendingnotification# ExamplesNotification@output<<"\n\nPENDING:\n\t"@output<<notification.pending_examples.map{|example|example.full_description+" - "+example.location}.join("\n\t")enddefdump_failuresnotification# ExamplesNotification@output<<"\nFAILING\n\t"# For every failed example...@output<<notification.failed_examples.mapdo|example|# Extract the full description of the examplefull_description=example.full_description# Extract the example location in the filelocation=example.location"#{full_description} - #{location}"end.join("\n\n\t")enddefdump_summarynotification# SummaryNotification@output<<"\n\nFinished in #{RSpec::Core::Formatters::Helpers.format_duration(notification.duration)}."enddefclosenotification# NullNotification@output<<"\n"endend

In the new dump_failures method we loop through every failed example. Then, we extract the description and the location of the failed example and we build a string that we append to the output. After this change, the output will look like this:

Next thing, how do we add the error messages underneath every failing spec? Lets expand the dump_failures method just a bit.

classGroupingFormatterRSpec::Core::Formatters.registerself,:dump_pending,:dump_failures,:close,:dump_summary,:example_passed,:example_failed,:example_pendingdefinitializeoutput@output=outputenddefexample_passednotification# ExampleNotification@output<<"."enddefexample_failednotification# FailedExampleNotification@output<<"F"enddefexample_pendingnotification# ExampleNotification@output<<"*"enddefdump_pendingnotification# ExamplesNotification@output<<"\n\nPENDING:\n\t"@output<<notification.pending_examples.map{|example|example.full_description+" - "+example.location}.join("\n\t")enddefdump_failuresnotification# ExamplesNotification@output<<"\nFAILING\n\t"# For every failed example...@output<<notification.failed_examples.mapdo|example|# Extract the full description of the examplefull_description=example.full_description# Extract the example location in the filelocation=example.location# Get the Exception messagemessage=example.execution_result.exception.message"#{full_description} - #{location}#{message}"end.join("\n\n\t")enddefdump_summarynotification# SummaryNotification@output<<"\n\nFinished in #{RSpec::Core::Formatters::Helpers.format_duration(notification.duration)}."enddefclosenotification# NullNotification@output<<"\n"endend

The only addition is on line 34 - we extract the result of the execution of the example, then we get the message of the exception that RSpec raised when the example failed. Now, lets test it:

This is all good, but you can see that the text alignment is broken a bit. If you look at the picture at the beginning, you will notice that the exceptions should appear indented underneath the description of the failing example. Lets fix this.

classGroupingFormatterRSpec::Core::Formatters.registerself,:dump_pending,:dump_failures,:close,:dump_summary,:example_passed,:example_failed,:example_pendingdefinitializeoutput@output=outputenddefexample_passednotification# ExampleNotification@output<<"."enddefexample_failednotification# FailedExampleNotification@output<<"F"enddefexample_pendingnotification# ExampleNotification@output<<"*"enddefdump_pendingnotification# ExamplesNotification@output<<"\n\nPENDING:\n\t"@output<<notification.pending_examples.map{|example|example.full_description+" - "+example.location}.join("\n\t")enddefdump_failuresnotification# ExamplesNotification@output<<"\nFAILING\n\t"@output<<failed_examples_output(notification)enddefdump_summarynotification# SummaryNotification@output<<"\n\nFinished in #{RSpec::Core::Formatters::Helpers.format_duration(notification.duration)}."enddefclosenotification# NullNotification@output<<"\n"endprivate# Loops through all of the failed examples and rebuilds the exception messagedeffailed_examples_outputnotificationfailed_examples_output=notification.failed_examples.mapdo|example|failed_example_outputexampleendbuild_examples_output(failed_examples_output)end# Joins all exception messagesdefbuild_examples_outputoutputoutput.join("\n\n\t")end# Extracts the full_description, location and formats the message of each example exceptiondeffailed_example_outputexamplefull_description=example.full_descriptionlocation=example.locationformatted_message=strip_message_from_whitespace(example.execution_result.exception.message)"#{full_description} - #{location}\n#{formatted_message}"end# Removes whitespace from each of the exception message lines and reformats itdefstrip_message_from_whitespacemsgmsg.split("\n").map(&:strip).join("\n#{add_spaces(10)}")enddefadd_spacesn" "*nendend

In the example above we took the extra step to format the error messages nicely. Basically, we split the exception message on a new-line character, we remove all the whitespace and we rejoin the pieces with a newline between them and add 10 spaces at the beginning of the message (for the indentation). Now, the output will look like this:

And voila, the formatter is working as supposed. Or, is it? :) Lets add some colors! Adding colors is really easy, we just need to require the ConsoleCodes module. The ConsoleCodes module provides helpers for formatting console output with ANSI codes, for example colors and bold. So, the final version of our GroupingFormatter is:

require'rspec/core/formatters/console_codes'classGroupingFormatterRSpec::Core::Formatters.registerself,:dump_pending,:dump_failures,:close,:dump_summary,:example_passed,:example_failed,:example_pendingdefinitializeoutput@output=outputenddefexample_passednotification# ExampleNotification@output<<RSpec::Core::Formatters::ConsoleCodes.wrap(".",:success)enddefexample_failednotification# FailedExampleNotification@output<<RSpec::Core::Formatters::ConsoleCodes.wrap("F",:failure)enddefexample_pendingnotification# ExampleNotification@output<<RSpec::Core::Formatters::ConsoleCodes.wrap("*",:pending)enddefdump_pendingnotification# ExamplesNotificationifnotification.pending_examples.length>0@output<<"\n\n#{RSpec::Core::Formatters::ConsoleCodes.wrap("PENDING:",:pending)}\n\t"@output<<notification.pending_examples.map{|example|example.full_description+" - "+example.location}.join("\n\t")endenddefdump_failuresnotification# ExamplesNotificationifnotification.failed_examples.length>0@output<<"\n#{RSpec::Core::Formatters::ConsoleCodes.wrap("FAILING:",:failure)}\n\t"@output<<failed_examples_output(notification)endenddefdump_summarynotification# SummaryNotification@output<<"\n\nFinished in #{RSpec::Core::Formatters::Helpers.format_duration(notification.duration)}."enddefclosenotification# NullNotification@output<<"\n"endprivate# Loops through all of the failed examples and rebuilds the exception messagedeffailed_examples_outputnotificationfailed_examples_output=notification.failed_examples.mapdo|example|failed_example_outputexampleendbuild_examples_output(failed_examples_output)end# Joins all exception messagesdefbuild_examples_outputoutputoutput.join("\n\n\t")end# Extracts the full_description, location and formats the message of each example exceptiondeffailed_example_outputexamplefull_description=example.full_descriptionlocation=example.locationformatted_message=strip_message_from_whitespace(example.execution_result.exception.message)"#{full_description} - #{location}\n#{formatted_message}"end# Removes whitespace from each of the exception message lines and reformats itdefstrip_message_from_whitespacemsgmsg.split("\n").map(&:strip).join("\n#{add_spaces(10)}")enddefadd_spacesn" "*nendend

As you can see, we are using the ConsoleCodes.wrap method which wraps a piece of text in ANSI codes with the supplied code in the arguments. You can now test our new colored formatter:

This works alright. But, requiring your formatter every time you run your specs is boring.

Meet .rspec

RSpec’s documentation says that RSpec reads command line configuration options from files in three different locations:

Local: ./.rspec-local - This file should exist in the project’s root directory. It can contain some private customizations (in the scope of the project) of RSpec and should be ignored by git.

Project: ./.rspec - This file should exist in the project’s root directory. It usually contains public project-wide customizations of RSpec and is usually checked into the git repo.

Global: ~/.rspec - This file exists in the user’s home directory. It can contain some personal customizations of your RSpec and is applied to every project where RSpec is used on your machine.

So, we can add a .rspec file in our project’s folder with the following contents:

--color
--require ./grouping_formatter.rb
--format GroupingFormatter

RSpec will read this file every time we run our specs, so this means that we can run our specs without specifying these options in the rspec command:

rspec some_spec.rb

This will now work with our new formatter.

Using it in a Rails app

Lets integrate our new formatter in a Rails application. Using the formatter in your Rails application is done in two steps:

The formatter class must either be in Rails’ autoload path, or manually required in the spec_helper. My personal preference is to require it manually because it’s more verbose.

In the RSpec.configure block in the spec_helper, you need to register the formatter to RSpec. This is done by:

config.formatter=NameOfTheClass

or, in our case:

config.formatter=GroupingFormatter

That’s it. Now when you run your specs the new formatter will be used by RSpec.

Outro

I hope you found this (quite long) post informative and interesting. If any of you wrote your own RSpec formatters, please, share them with me and the others in the comments - I am very curious to see what you’ve come up with. Thanks for reading to the very end!