Setting up Puppet module testing from scratch: Part II, Beaker for module testing

In our previous post we covered the py_puppetlabs_spec_helper, a kind of front-end to a bunch of tools including puppet-syntax, puppet-lint, and rspec-puppet.

Today we will be looking at how to set up the Beaker framework for module testing. Beaker is the Puppet community’s preferred acceptance test harness.

Before proceeding, I must call out some of the materials that I found useful when I learnt Beaker, especially Liam J. Bennett’s great three part series, Testing puppet with Beaker, Part 2: the Windows Story, and Part 3: testing roles. There is also a great 1-hour presentation by Puppet’s David Schmidt online here, which covers a lot of the material we look at in this post and in the previous post (the Beaker-related material starts at around 40 minutes).

Once again, my focus is on helping new people to quickly stand up the Beaker framework, while assuming as little as possible by way of Ruby and any other prior knowledge; I am less concerned about writing a proper tutorial on writing Beaker and Serverspec tests. And as before, I use the real life example throughout of the Spacewalk module we are developing.

Installing and configuring Beaker

Prerequisites

To get started, we’ll need to ensure we have our prerequisites installed. In addition to Ruby Gems and Bundler, which were discussed in Part I, we’ll also need to install Vagrant. I assume that my readers have used Vagrant before; if not, it’s well worth spending 15 minutes doing the Vagrant tutorial.

Otherwise, let’s continue.

Gemfile additions

As we discussed in Part I, the Gemfile is a file that specifies Ruby dependencies to be installed in our ‘bundle’ by Bundler, a Ruby app that manages dependencies in Ruby projects.

To install Beaker, we will add a new system testing Gem group to the Gemfile:

To understand these additions, be aware that Beaker really has two parts, Beaker itself, and Beaker-rspec, a shim that connects Beaker to Rspec and Serverspec. Meanwhile, the beaker-puppet_install_helper is a helper library that takes care of installing Puppet in all of its various open source and PE versions.

If you are wondering why we add a Gem group :system_tests distinct from the :tests Gem group, this is mainly so that our Travis CI build system, which we discuss in Part III, won’t need to install all the system test related Gems (unless it is going to actually run the system tests, which is often not the case).

Rakefile

The :beaker and :beaker_nodes tasks

You may have noticed in Part I of this series that the :beaker and :beaker_nodes tasks were placed in the list of Rake tasks by the puppetlabs_spec_helper. Well, the :beaker task will run all of the specs in spec/acceptance. I rarely use that one. The other one, :beaker_nodes, essentially just lists the files I have in the nodesets directory:

So, you’ll probably find that you don’t actually need the Beaker-related tasks from the puppetlabs_spec_helper, but it’s still good to know what they are.

So the take away point is, when configuring Beaker, you don’t really need to touch the Rakefile, and I’ve covered the config you might find in here in case you’re interested.

The spec helper acceptance

The acceptance test spec helper file, however, is very important.

As we mentioned in Part I, the spec helper is a file that is used by convention to configure Rspec, and we will already be using that file to configure Rspec-puppet. So for Beaker, a special spec helper file is typically used to configure Beaker-rspec, and it lives at spec/spec_helper_acceptance.rb.

Digression: The Beaker docs

Before proceeding I’d like to note some important sources of Beaker documentation that will help in understanding the spec/spec_helper_acceptance.rb file.

The README of Beaker-rspec, where an (at the time of writing, out of date) set up tutorial can be found there.

The Beaker::Host object and the nodesets

It’s also important to be aware that the Beaker DSL makes available to Rspec an array named hosts that contains Beaker::Host objects, a.k.a. “systems under test” or SUTs. (Only follow the link if you know Ruby well.)

The SUTs themselves are initialised according to the HOSTS Hash in the nodesets files that we copied earlier. For example, our the CentOS 7 node set:

Firstly, note that the keys of the HOSTSHash in our node sets correspond to elements in a hostsArray in our spec helper.

Secondly, it is normal to have only a single host defined in the nodesets; you’ll only ever see multiple hosts in here in multi-node configurations that use Beaker DSL in multi-node configurations.

Puppet install helper

A perceptive reader may have noticed that the Beaker-rspec documentation includes an example spec helper that includes the following lines:

# Install Puppet on all hosts
hosts.each do |host|
on host, install_puppet
end

Don’t do this, because we now have the Puppet install helper to do this for us, which not only installs Puppet, but also sorts out for us how to install all the various versions of Puppet. I’ll have more to say about this helper below. For now, just be aware that we’re going to replace the 4 lines above with a single line:

run_puppet_install_helper

Putting it all together

Ok, so we have all of the pieces we need, so here is the code for our spec/spec_helper_acceptance.rb:

So we require the beaker-rspec and beaker/puppet_install_helper, we run the install helper, then in our RSpec.configure block we define a variable for our project root directory, we configure RSpec’s output, and then declare a before :suite hook (set up code that is run once before all of our tests).

Notice that our hosts.each loops through the hosts, and then inside this loop we copy the module to the host and on that host we call puppet module install puppetlabs-stdlib.

Why loop through an array if we already know that it has only one element? Well, convention. We could equally refactor as:

A simple test case

The apply_manifest method and the test for idempotence

Although my intention is not to write a tutorial on writing Beaker tests, I should mention in passing the apply_manifest method from the Beaker DSL, which is a wrapper around the puppet apply command.

Every suite of Beaker tests will at some point need to apply some Puppet manifest code, and it is typical for the first test case to apply a Puppet manifest using apply_manifest and passing it :catch_failures => true, and this will be followed by a second test case that applies the manifest again, and expects the exit code to be zero.

In the example of our Spacewalk module, we’ll write these test cases as:

require 'spec_helper_acceptance'
describe 'spacewalk::server' do
let(:manifest) {
<<-EOS
include spacewalk::server
EOS
}
it 'should apply without errors' do
apply_manifest(manifest, :catch_failures => true)
end
it 'should apply a second time without changes' do
@result = apply_manifest(manifest)
expect(@result.exit_code).to be_zero
end
end

Extending the test case with Serverspec

It’s likely that remainder of your test cases will use the Serverspec extensions, and as such I’d refer the reader to the Serverspec tutorial.

In the case of our Spacewalk module, we’ll expect that our server will be at minimum listening on ports 80 and 443. So we add to our describe two more:

describe port('80') do
it { is_expected.to be_listening }
end
describe port('443') do
it { is_expected.to be_listening }
end

Running the tests

Specifying the Puppet version

Now, here’s a big gotcha. By default, the Puppet install helper will install the latest Puppet 3.x, and not the latest Puppet 4.x! At the time of writing, that means we’ll get Puppet 3.8.6 instead of Puppet 4.4.2.

And wham, followed by a second big gotcha: to set the Puppet version to Puppet 4.4.2, we need to understand that the environment variable $PUPPET_INSTALL_VERSION is overloaded when used in conjunction with $PUPPET_INSTALL_TYPE. It is explained here in the docs.

If you want to install Puppet 3.x, that’s easy: set $PUPPET_INSTALL_VERSION to the version you want, and ensure that $PUPPET_INSTALL_TYPE is unset. In other words:

$ export PUPPET_INSTALL_VERSION=3.3.2
$ bundle exec rake beaker

If you want to install Puppet 4.x, however, the $PUPPET_INSTALL_TYPE must be set to agent, and then $PUPPET_INSTALL_VERSION takes on a new meaning, as the version of the Puppet Agent. Thankfully there’s a matrix converting Puppet versions and Puppet Agent versions here. So if I want Puppet version 4.4.2, then I need Agent version 1.4.2. Confused yet?

Specifying the node set

Specifying the node set is much easier. Just set the variable $BEAKER_set to the node set:

$ export BEAKER_set=centos-72-x64
$ bundle exec rake beaker

If left unset, it will default to default (i.e. it will use whatever is specified in spec/acceptance/nodesets/default.yml.

Other important environment variables

Another important environment variable is $BEAKER_destroy. If you set this to no, the VM will not be destroyed after the test/s.

This is really useful for debugging, and perhaps the best part, your acceptance tests double as a convenient way of quickly spinning up a VM that is ready-configured by Puppet. So if I ever want a Spacewalk server to play with, I just run my spec test, and voila!

Understanding the output

I’ll need to truncate the output, as there will always be a lot of it.

To begin, Rspec is called, and it is quite common to call the rspec command directly:

$ bundle exec rspec spec/acceptance/spacewalk_server_spec.rb

At the time of writing, the latest version of Serverspec is emitting some warnings that I am not interested in:

If you know how to silence these then feel free to let me know in the comments! One of the fun parts of working with Beaker is that the maintainers of Serverspec don’t allow us to raise issues, but will only accepts PRs that resolve the issues you’ve found, and I don’t have time right now.

Now I’m going to truncate a bunch of output, as Beaker sets up SSH keys; configures the /etc/hosts file; reconfigures the SSH daemon; and a bunch of other stuff, before it comes around to installing Puppet:

The line “should apply without errors” is the continuation of the Rspec documentation-formatted output, and it’s telling me that the test passed. (If it’s not telling you that yet, don’t worry; you’ll get used to Serverspec’s output format.)

Rspec tells me that my 4 examples took 13 minutes and 6 seconds and all of them passed.

Well, that’s it for part II of the series.

4 Comments

Brad
on December 7, 2016 at 1:44 am

This has been a useful intro, many thanks.

One thing I still don’t understand is how you are including the class like so:

let(:manifest) {
<<-EOS
include spacewalk::server
EOS
}

I understand that the copy would create a directory on the target node called spacewalk and that inside it you presumably have a class called server? Would have been nice to see it as I am struggling to get beaker to find a test class right now.

Can you include the manifest file and maybe some comments on how beaker finds it?

The ‘let manifest’ block results in a temporary file being created with one line ‘include spacewalk::server’. That temporary file is then applied via apply_manifest, which has the same effect as ‘puppet apply’.

The module is copied to the default module path as appropriate for the version of Puppet you told it to use. If that directory is ‘/etc/puppetlabs/code/modules’ then your module will appear in ‘/etc/puppetlabs/code/modules/spacewalk’ and you’ll have the server class in ‘/etc/puppetlabs/code/modules/spacewalk/manifests/server.pp’.

If your class is not being found, I guess it’s time to set $BEAKER_destroy to ‘no’ so that you can log in and debug this further.

Some really good info, thanks!
Do you know how you wire hiera into beaker? I’ve searched high and low but haven’t seen anybody post anything about it 🙁
I’m currently on 4.2.2 so no in-module hiera, this beaker effort is in part to ensure my move to something 4.5+ will be smooth so I need to have tests succeed first.
Any help or links to get me going would be much appreciated.
Thank you!