December 21, 2011

This article is written by Brandon Burton, who can mostly be found posting lolcats and retweeting @solarce, though he occasionally posts interesting links to things sysadmin, devops, and unix.

As systems administrators, we all know that it's not in production until it's monitored, but this isn't always as simple a rule to live by as it may sound. Not all web applications, for example, are easily monitored through traditional monitoring solutions such as Nagios, Zenoss, or various commercial tools. These tools tend to take a "curl | grep" style monitoring, or they may support somewhat more complex POSTing of XML or JSON data and validation of the returned data. But often the most key parts of applications being deployed into production involve complex browser interactions and behaviors - AJAX, or some other session or transaction that traditional monitoring frameworks don't have an easy way to accommodate.

Enter Selenium. Selenium is a mature and robust framework for doing complex interactions with web applications. It originated as a tool at the consulting company ThoughtWorks as a way to do testing against web applications by driving a web browser. Since its release, it has seen the development of numerous tools, including browser plugins to make it easy to develop Selenium tests quickly and easily, language bindings to write tests in pretty much every major language, and tools to run many browsers across many operating systems, in parallel.

Additionally, services, such as BrowserMob and Sauce Labs, have grown around the Selenium ecosystem to help you do testing and monitoring in a scalable and offsite fashion. It is these services that we'll focus on utilizing in this blog post.

So what does all this mean? It means that we have a mature and robust toolset that we can utilize and perform testing and monitoring of the complex web applications that we are deploying into production.

Getting started

So how do we get started? My preferred method is to begin by developing tests locally. You can use the Selenium IDE, but for this example I'll show a Firefox extension called Sauce Builder which makes it a snap to build and run your first test locally.

To get started you'll need Firefox installed, then go to the Sauce Builder download page and walk through getting the extension installed.

Once you've got the Sauce Builder extension installed, it is time to build our first test.

I'm going to walk you through building a test to search for jelly beans on Amazon.

Open Firefox

Click on Tools -> Sauce Builder

Enter ''amazon.com'' in the Start Record prompt and click Go

Enter ''jelly beans'' for the search term

Click Go

Click on the first search result, for me this was '''Kirkland Signature Jelly Belly Jelly Beans 49 Flavors (4 Lbs)'''

Go back to the Sauce Builder window and click Stop recording.

Now that we've recorded a test, we should save it for safe keeping. Click File -> Save or Export -> Choose HTML as the format and name it, then click Save.

As you can see from the test we've recorded. The test is composed of a series of actions and each action will have one or more options associated with it.

Obviously this is a pretty simple test and you could do a lot more with it, including go through adding it to a cart, checking out, and buying the order. But for the purposes of getting started, it's a good place to stop.

The next thing we want to do, since our focus is on monitoring, is add some verification steps to each page load. This step is crucial in making our test doing the same kind of checking that your traditional curl URL | grep STRING style monitoring did, but now it's integrated into our browser-driven mode of execution.

Go to the Sauce Builder window

Mouse over the second step and choose New step below

Select the new step

Choose edit action

Select the assertion option

Choose page content

Choose assertText

Click Ok

Choose locator and enter ''link=Your Amazon.com'

Click Ok

Choose equal to and enter the string ''Your Amazon.com''

Click Ok

Click on Run and Run test locally

The test should run successfully, if it does not, then you may want to click on locator and choose Find a different Target and use the tool to select the element you're asserting text with.

This is a critical step as the assertions are somewhat brittle and must be maintained as your application changes over time. For more details, see help on choosing good locators.

Using Sauce Labs for Testing

Now that you've gotten your test running locally and you've added some assertions to make the test useful for monitoring, it is a good idea to run the test externally. As previously mentioned, the Sauce Labs folks run a service to run your tests in the Cloud, and they are nice enough to offer a free plan that gives you 200 "execution" minutes per month and the ability to run your tests under multiple browsers and operating systems with ease. Plus you'll get your jobs stored, logs, screenshots, and a video recorded of the whole test for later review and analysis. So now that you're thinking "where do I sign up?!"

You'll end up with a Job URL that looks something like https://saucelabs.com/jobs/6f4629f04dad85cd7803d8049ec00888 (which I've made public, since there is nothing private in it.)

Review the details of the test, as you can see, you get the following for each test

Platform

Start and End Times

Duration

Status

Break down of each Selenium command that's executed

Screenshot of the final page of the test

Video recording of the whole test run.

At this point you've successfully executed a test on Sauce Labs. I recommend you review the following to get a full idea of Sauce Labs features, which includes being able to use it programmatically from various languages, which is beyond the scope of what I'm covering this post.

Using BrowserMob for monitoring

So you've succeeded in getting your test run locally, you've run it externally in the "Cloud", and now you're thinking "wasn't I promised I could use this for monitoring?". Yes, you were, and that's where BrowserMob comes in.

While BrowserMob's primary product is focused on load testing, they've also built a great monitoring product and that's what we'll using to get our monitoring up and running.

BrowserMob is kind enough to offer a free plan, so let's start with getting signed up.

Confirm the contact name and email, it will default to what you registered with

Click Create

Now let's set up a monitoring job.

Click on Monitoring

Click on Schedule

Give the job a name

Select a Frequency * With the free account, you can run a simple test every 12 hours, for higher frequency or more complicated tests, you'll need to purchase a paid account.

If you want to do an alert, click create

Select a location

Select your notification preference

Click Activate Now

The job will be scheduled and will run at the next internal after the minute the job was created.

Since you just signed up for a trial, you can get the test to run a bit sooner, but only a couple times, so we'll do that now, so we can see what it looks like.

Click Edit next to the test

Change Frequency to 10 minutes

Click Save and Activate

Set a timer for 12 minutes and wait, once it is done, we'll review what things look like. * Once you're done with this, you may want to revert to every 12 hours so that when you're trial expires you won't be over your credits, or just pause/delete the monitoring job.

So now that that test has run, let's take a look at what it looks like.

Click on Dashboard

Mouse over the name of the job and click on the URL, it should look like something like this: https://browsermob.com/monitoring/view/{some_id_here}

You should see a chart that defaults to 1 day and shows you each test, with a bar showing each data point, based on the overall time it took to run the test. * This gives you some quick insight to how performance (as measured by execution time) is doing over time.

You can drill into each data point, and you'll get a waterfall style break down of each test run: how long each element of the page took to load, etc.

Below is a screenshot of a test that has run for a few days.

So a couple tips on how you can use custom stuff from BrowserMob's API to make your tests that much more effective.

Setting variables.

Since the BrowserMob scripts are written in JavaScript, doing variables is as simple as doing var zipcode = '90210'

Getting back data from a webpage.

I've only ever used this to get back the whole response from a page and use it as is, so you'd need to break out a bit of your own JS-fu if you want to use part of a response, but here's how I did it. The code below also shows using a previously declared variable in your request.

As a challenge and a way to motivate people to contribute and give feedback, the 5 most interesting tests that people submit as pull requests on Github, I will send them a package of stickers, including SysAdvent, Github, Riak, and more!

I hope you've found this post to be informative and would love feedback via email or Twitter on how you do end up using any or all of the services in this post.