December 17, 2010

Developers love tests: testing, quality, and inspection tools. Modern testing
practices yield automated tests for code and means to run them constantly while
developing an application. When the app hits production, the operations team
often have a different set of tools for monitoring the health of everything
from disk space to requests per second to service health. To get the tested
code into the well-monitored production environment, you need to deploy it, and
that's where smoke testing comes in.

But What is Smoke Testing?

Smoke testing is non-exhaustive software testing aimed at ensuring that the
most crucial functions of an application meet expectations, but skipping
on the finer details. Smoke testing your deployments simply means having a
test suite you can run very quickly against your production environment just
after your deployment to make sure you've not broken everything.

You can write your smoke tests with whatever tool you choose, but I'm going to
show some simple examples using the Ruby tool: Cucumber.
I've found Cucumber useful for smoke tests as it makes it very simple for
everyone involved, including project manager and business stakeholders, to
understand what is being tested. Cucumber is useful because smoke tests need to
be very fast and targeted to be useful, which means making judgements about
what is critical, which requires a common language for communicating what is
critical and how it should be tested.

I'm going to run this example using jruby in order to use another great tool,
Celerity, which is a jruby wrapper around a
headless browser. You don't have use Celerity to do this; lots of people use the
Webrat library to make web requests
instead. I like Celerity because it can execute the javascript on a page,
meaning you can test more complex applications and more complete functionality.

An Example

I'm going to show a real world test from
FreeAgent, which checks that the login box
appears when you click the login button on the homepage. This would be just one
part of a larger smoke test suite but is hopefully a good simple but
non-trivial example.

First we need a few dependencies. Here are the instructions for installing on a recent version of Ubuntu, although any system you can run jruby on should be fine.

Next we create a cucumber feature file in features/homepage.feature, which describes in a structured but human readable format exactly what we're testing.

Feature: Homepage
So we can keep existing users happy
Visitors to the site
Should be able to login
Scenario: Check login box appears when login button is clicked
Given I'm on the homepage
When I click the login button
Then I should see the login box

You don't have to use cucumber for writing smoke tests, but I find it useful
because I can easily discuss what is being tested with other non-developers
simply by sharing the feature file contents (above).

Next we write the actual code that makes the test work in
features/steps/homepage.rb. I've included everything in one file for
simplicities sake but in a larger example you would probably separate out
utility functions and configuration from the step code. For a larger test suite
you'll also find that you can reuse many steps by passing in arguments from the
features files.

require 'rubygems'
require 'celerity'
require 'rspec'
BROWSER = Celerity::Browser.new
TIMEOUT = 20
# this is a simple utility function I use to find content on a page
# even if it might not appear straight away
def check_for_presence_of(content)
begin
timeout(TIMEOUT) do
sleep 1 until BROWSER.html.include? content
end
rescue Timeout::Error
raise "Content not found in #{TIMEOUT} seconds"
end
end
Given /^I'm on the homepage$/ do
BROWSER.goto("http://www.freeagentcentral.com")
end
When /^I click the login button$/ do
check_for_presence_of "Log In"
BROWSER.div(:id, "login_box").visible?.should == false
BROWSER.link(:id, "login_link").click
end
Then /^I should see the login box$/ do
BROWSER.div(:id, "login_box").visible?.should == true
end

To run the feature we've just created just run the following command in the
directory where you created the files - cucumber looks for a 'features' directory.

jruby -S cucumber

This should output test results showing what ran and whether it passed:

Feature: Marketing Site
So we can keep existing users happy
Vistors to the site
Should be able to login
Scenario: Check login box appears when login button is clicked # features/homepage.feature:6
Given I'm on the homepage # features/steps/homepage.rb:1
When I click the login button # features/steps/homepage.rb:5
Then I should see the login box # features/steps/homepage.rb:11
1 scenario (1 passed)
3 steps (3 passed)
0m8.411s

Cucumber provider a number of other output formats that might also
be useful (html, etc), and Cucumber Nagios has an
output formatter for the nagios plugin format, too:

Start Simple

A more complex example might step through a multi-stage form to test the
purchase of a product, or it could conduct a series of searches to check a
search index has been populated. Smoke testing and Cucumber are not just for
web apps either. You should be testing all of your important services and
systems. Unlike lower level testing you want to touch as many individual parts
of the app as possible, including testing that third party API's or parts of
your infrastructure are up and running. You definitely don't want to
mock out your database calls and
then find that the app actually fails due to a problem with the database coming
back up after a deployment.

Once you have a smoke test you can run manually each time you deploy, you can take
the next step: automation. Running the smoke tests automatically as part of
whatever deployment mechanism you have might be useful, logging the results or
integrating the output into a reporting tool like nagios might work well for
your team, too. Automated deployment followed by a smoke test failure could
invoke an automated rollback.

Deployment actions are still a time of higher-than-average risk for most projects,
and strong smoke tests are important regardless of your deployment frequency or
project size. If you have only a few machines, then smoke testing might tell
you that you need to roll back a deployment immediately. If you have a larger
infrastructure, then smoke testing a newly upgraded application server before
putting it back into a load balancer rotation could save you from any ill
effects in production at all.