Search form

The Next Generation of Puppet Module Testing

Breadcrumb

We’ve written severalblogposts about testing modules. There’s good reason for this: testing is an increasingly important part of module development here at Puppet Labs. Comprehensive module test suites help us avoid regressions and support a wide range of Puppet and operating system versions. Like everything we do, we want the work that we’ve put into testing to be of as much benefit to the Puppet community as possible.

Despite all this prior work, module testing has remained somewhat painful. There’s a lot of boilerplate to write and, for complex modules, there can be a number of dependencies which need to be setup correctly. These dependencies make life especially difficult when trying to configure automated testing—every module would need its own testing script.

To resolve this, we’ve been working on a common set of testing helpers which any module author can use to abstract away that common boilerplate. These helpers take the form of a Ruby gem called puppetlabs_spec_helper. The source code for this gem is available on Github.

This gem provides a few things:

A common helper to reduce the boilerplate needed to use rspec-puppet, a unit-testing framework for puppet modules.

A set of Rake tasks that handle common testing and development tasks including:

Installing and cleaning up test dependencies

Building a module package for the Puppet Forge

Verifying best practices with puppet-lint

The rest of this blog post assumes you’re at least somewhat familiar with module testing and rspec-puppet. If you’re not, I recommend you read the blog posts linked above and then come back here.

A Simple Module

To demonstrate the puppetlabs_spec_helper gem, we’re going to add tests to a simple module that installs apache. Let’s start by creating that skeleton module. Create a file in modules/apache/manifests/init.pp with the following contents:

This manifest is designed to demonstrate both important situations for writing module spec tests:

Are there any conditionals? If so, tests should be written to verify that all branches are correctly taken.

Are there any forwarded parameters? It’s important to ensure that your code accurately passes parameters to contained classes and types.

Testing the Module

Now that we have a module, it’s time to test it. First, let’s make sure puppetlabs_spec_helper is available to us. At the time of this writing, the latest version is 0.2.0.

% gem install puppetlabs_spec_helper

We’ll be adding several more files to the project. If your module was created using the Puppet Module Tool, it may already contain some of these files; you will need to replace them with the versions described here in order for the puppetlabs_spec_helper to work correctly. If your module does not contain these files you will need to create them manually.

This is where using puppetlabs_spec_helper diverges from the traditional rspec-puppet setup. If you’ve tested a module with rspec-puppet before you will notice a new file—.fixtures.yml. This file is used by the common rake tasks to automatically install dependencies for test runs. For our example, its contents should be:

This defines a testing environment with two modules: one called ‘stdlib’ fetched from a Github repository, and one called ‘apache’ (our module) which will be a symlink to the current source directory.

Another difference between traditional rspec-puppet code and the puppetlabs_spec_helper gem is the boilerplate. Both the Rakefile and spec/spec_helper.rb are provided by the gem, and should be two-liners in your module:

It is possible to add additional functionality to these files if you desire. Whenever adding new functionality you should try to decide if it truly is module-specific. If it could be useful to other modules the best option is to open a pull request against puppetlabs_spec_helper.

The remaining three tasks allow you to handle setup, teardown, and testing as individual steps.

Writing the Test Code

Finally it’s time to write the test cases for our module. As discussed earlier, our example module is designed to demonstrate the important testing points. We will need to write several tests to cover them:

If a package name is provided, it should be used

If a package name is not provided AND we are on a supported osfamily, a default name should be used.

Since there are two supported osfamilies, we should write tests for both of them.

If a package name is not provided AND we are not on a supported osfamily, the catalog compilation should fail.

This is a total of four fairly simple test cases. Writing these tests is exactly the same as if you were using rspec-puppet without the puppetlabs_spec_helper gem. Their implementation will go in apache_spec.rb:

As you can see, the spec task automatically cloned the puppetlabs-stdlib repository before running the tests.

Continuous Integration

This automatic handling of module dependencies makes it very easy to test modules in continuous integration systems. We do this ourselves using Jenkins for a number of modules.

Using Bundler

While not strictly required, bundler makes it easy to test your module against several versions of Ruby and Puppet. Most CI systems expose test axes as environment variables, and a specially written Gemfile will allow you to use those environment variables to install the correct versions of dependencies. The Gemfile used by Puppet Labs for our modules is:

Aside: Gemfiles and the Puppet Module ToolIn our modules, we name our Gemfiles .gemfile instead of Gemfile. Dotfiles are automatically ignored when packaging with the Module Tool. We recommend this practice in order to avoid clutter on end-users’ systems, but it is not strictly required.

Testing on Jenkins

If you are using the Gemfile above, your jenkins configuration should contain an axis called PUPPET_VERSION with the values being the Puppet versions you wish to use for testing.

In our environment, we use RVM to isolate test runs in their own gemsets. This is optional, and if you have a different method of isolating test runs you can safely remove the references to rvm from the test script below. $ruby is a Jenkins axis containing the various Ruby versions we test against.

This script uses the Gemfile described in the previous section to populate a temporary unique gemset with the testing dependencies. It then runs the tests and cleans up the gemset.

Testing on Travis CI

If you do not have access to a Jenkins installation, Travis CI provides an environment that is compatible with testing Puppet modules. Unlike Jenkins, Travis will automatically handle setting up a Ruby environment and running bundler on the .gemfile. Thus a special script is not needed; we can instead use a simple configuration file. Travis requires this file be named .travis.yml.

A detailed explanation of Travis configuration files is outside the scope of this article. If you’re unfamiliar with the Travis Yaml configuration files you should read the documentation.

Conclusion

Puppetlabs_spec_helper is part of our larger goal of encouraging testing on Puppet modules. The standardized rake tasks and common boilerplate enable module authors and contributors to easily write and run tests for Puppet code and the automation features make it easy to run tests in CI environments. Our hope is that as the tooling and process around module testing improve we will see tests become a more ubiquitous part of module development both here at Puppet and in the community at large.