I often find it difficult to understand a test. This can happen when the test was written by someone else, or even when I wrote it myself mere months ago. Usually the heart of the test is simple and each line of code is easily understood, but the context and setup can be extensive and located far away.

I’ve started using ASCII tables to define the initial test data. These tables are both concise and easy to understand, and that leads to simpler tests that are easier to maintain.

For context, at Indiegogo we use rspec, jasmine, and other testing frameworks in the Behavior-Driven Development category. Our tests will often specify nested contexts that enumerate possible states before adding our expectations. A typical test looks something like this:

describe '#my_method' do
context 'when foo is true' do
let(:foo) { true }
context 'and bar is true' do
let (:bar) { true }
it 'returns W' do
expect(my_obj.my_method).to eq('W')
end
end
context 'and bar is false' do
let(:bar) { false }
it 'returns X' do
expect(my_obj.my_method).to eq('X')
end
end
end
context 'when foo is false' do
let (:foo) { false }
context 'and bar is true' do
let (:bar) { true }
it 'returns Y' do
expect(my_obj.my_method).to eq('Y')
end
end
context 'and bar is false' do
let (:bar) { false }
it 'returns Z' do
expect(my_obj.my_method).to eq('Z')
end
end
end
end

Unfortunately, these structures often explode into huge files with thousands of lines of code. It’s pretty difficult to know the expected state for a line of code when the test data is initialized in multiple places hundreds of lines away.

Now, consider an alternative organization for the example above using ASCII tables:

The table provides a concise and easily-understood presentation of the various states and the expected result.

This is not a new idea. Ward Cunningham’s Fit: Framework for Integrated Test and Bob Martin’s FitNesse both show state and expectations in tables, but making tables with ASCII characters means that you can embed these in existing test files, yielding many of the benefits of Fit and FitNesse without adopting the whole framework.

RubyMine’s column selection mode helps but I’m hoping this offering will inspire you to create a RubyMine plugin that is column aware like the table editor in org mode. If you do, please share.

Are ASCII tables right for your test?

Scale is important when deciding if you should use an ASCII table or some other approach for establishing test data. An ASCII table is probably overkill if you are initializing just a few variables. Likewise, if your test is initializing 2 or more attributes with two or more states then consider what the initialization would be like if summarized in a table. Also consider organizing your data using arrays and hashes with white space to show structure.

Nested logic is difficult to get right the first time and really difficult to maintain. Here’s a simple technique that flattens nested logic into a single level case statement that is easier to get right and far easier to maintain. The technique has 2 parts: concatenate state, and case.

The goal is to transform a nested “if” into a flattened case statement.

if a == 'x' then
if b == 'y' then
print 'a=x, b=y'
else
print 'a=x, b!=y'
end
else
if b == 'y' then
print 'a!=x, b=y'
else
print 'a!=x, b!=y'
end
end

This is easier to understand. It is very easy to enumerate all permutations of the possible state components.

When you do this the initial implementation is easier. But the really big win is when you return to the code weeks, months, or years later. By then you’ve forgotten all of the interconnected logic that led you to the original implementation. You are forced to stare at the implementation and attempt to rebuild your mental model. With concatenate state, and case all of this is laid out in simple order.

How do you do it? First you bundle all of the state into a single variable. Then you drop into a case statement with all posible permutations of those states to find the code to execute.

Some languages let you have multiple value case expressions, with syntax something like:

In this contrived example the strings are just as verbose as their non-string expressions, so there is no win. But often the expressions are not so succinct but the strings can be as succinct as you make them, while hopefully communicating the status clearly.

Here’s a real life example pulled from some code I wrote a few years ago. The problem is to validate passwords in a user object on save. There are 3 attributes that are of interest: a password hash, a new password, and a new password confirmation.

The logic goes like this: We need either an existing password hash or a new password and matching password confirmation. If we have a new password or a new password confirmation, then we are attempting to change or set the password, and both items must be present and they must match. If neither are present, but we have an existing password hash, then we are valid, meaning we are not trying to change our password. If neither the new password or new password confirmation are present, and the password hash is blank, then we are invalid since all user records must have a password.

A first implementation might look something like this:

if password_hash.blank?
if new_password.blank?
errors.add('new_password', 'is required')
else
if new_password_again.blank?
errors.add('new_password_again', 'new password must be entered twice')
else
if new_password != new_password_again
errors.add('new_password_again', 'does not match your new password')
end
end
end
else
if new_password.blank?
if new_password_again.present?
errors.add('new_password', 'new password must be entered twice')
#else
# ok
end
else
if new_password_again.blank?
errors.add('new_password_again', 'new password must be entered twice')
else
if new_password != new_password_again
errors.add('new_password_again', 'does not match your new password')
end
end
end
end

That’s 27 lines of code (and 2 comment lines). And it is treacherous code. I think it works because I have a test suite and it passed. Without that test suite my confidence in this code would be near zero.

Let’s try cleaning it up a bit.

if password_hash.blank? && new_password.blank? && new_password_again.blank?
errors.add('new_password', 'is required')
end
if new_password_again.present? && new_password.blank?
errors.add('new_password', 'new password must be entered twice')
end
if new_password.present? && new_password_again.blank?
errors.add('new_password_again', 'new password must be entered twice')
end
if new_password.present? && new_password_again.present?
if new_password != new_password_again
errors.add('new_password_again', 'does not match your new password')
end
end

This is a little easier to read but no easier to ensure that it is correct.

This weighs in at 26 lines and 12 comments — still a pretty heavy method. However, this one carefully enumerates all the possible permutations of the state values — 4 state values, each with 2 possible values, for 16 possible states.

Of the 16 possible states, some are logically impossible. For example, you can’t say that the new password was specified and the new password confirmation was not specified but that the new password and the new password confirmation are the same — it can’t happen, so you can safely ignore that case.

The remaining states are either valid or invalid. When they are invalid some action is taken.

This code is as clear to me today as it was when I wrote it over 3 years ago.

Be careful when using AR models in migrations. Many of us know this and some know the workaround of defining your model in the migration, as documented in the aging Rails Recipes book by Chad Fowler in the recipe “Safely Use Models in Migrations.”

As the recipe specifies, you may need to refresh the model’s column data, with the “reset_column_information” class method.

I was caught by surprise when I did this on models that extended a base class using single table inheritance. My first try was to reset the column information of the base class.

Nothing complained. The data was simply not what I expected. After a while I decided to reset the column information for every sub class — voilà, success.

Tobias Lütke described a technique for moving data from one database to another using rake, active record, and yaml. Unfortunately his rake task is out of date and comments are closed on his article. An updated version of the rake task is available on github.

Selenium RC Fu is a fantastic system for testing Ruby On Rails applications. It is the blending of xUnit testing with selenium. Selenium is an amazing system that operates your browser as if a human were sitting there moving the mouse, pressing buttons and keys.

Selenium RC Fu is also a remarkable example of the power of open source. It’s Selenium remotely controlled by rails and ruby. You can learn more about it by viewing the slides for Full-stack webapp testing with Selenium and Rails presented by my colleagues Alex Chaffee and Brian Takita at the SDForum Silicon Valley Ruby Conference.

Now that you are excited about Selenium RC Fu, by law I must inform you that this wonderful testing tool comes with some costs. First, this is the daisy cutter of testing — problems will be detected, but it won’t be too specific about those problems. A failed selenium test will likely only tell you some expected text was not present on the page — you have to do some digging to discover the real problem.

It’s also slow. To be fair, a lot of software is running to do this testing.

So use selenium testing sparingly. A good strategy is to restrict selenium testing to “happy path” testing. These happy path tests become a compliment to other more focused and faster unit and integration tests.

Getting Started With Selenium RC Fu

Selenium RC Fu is hosted at rubyforge.org in the “pivotal.rb” project. It’s not well documented and a little hard to find. The primary documentation is the README File.

Many of the functions require a ‘locator’ argument. This needs to identify a single element on the page. Often the name or id of an element is sufficient, but you may need to use an XPath. Check the docs for more information on element locators.

These functions were sufficient for me. Look for more in vendor/plugins/seleniumrc_fu/lib/seleniumrc_fu/selenium_dsl.rb.

You can run your selenium tests individually or as part of a suite. Selenium RC Fu comes with a suite script in the vendor/plugins/seleniumnrc_fu/sample directory. Drop that in your test/selenium directory.

The selenium server must be running before running individual tests. You can start the server with the rake command rake selenium:server. Once the server is running you run your tests from the command line or via the suite.

You can also run the suite with the command rake selenium:test. This command is smart enough to start the server if it’s not already running. However, when the server is started by this mechanism its a little more work to stop it. You can stop the server with a simple ^c if it was started via the rake selenium:server command.

Make It So

That’s really all you need to build your tests but here are few more things that may help.

Site Diagram

A site diagram is handy for building your happy paths tests. There will be a lot of paths through the site and marking the paths on the diagram as you visit them is an easy way to track your progress. On the diagram you need to indicate both pages and the paths to those pages.

The later is really important. It’s the paths that you will be testing with your happy path selenium testing. Often there are buttons, links, or other controls such as check boxes that may touch the server and reload the current page, or just update the current page via ajax. Be sure to add these to your navigation diagram.

Here’s the diagram I used for my site.

Selenium IDE

You can code your tests by hand using Selenium RC Fu functions but it’s easier if you use Selenium IDE. This is a Firefox plugin that records your activity. That activity is available in various selenium dialects and can be pasted into your tests. Unfortunately, it does not have output for Selenium RC Fu. Nonetheless I still found it useful. I simply clicked through my application while Selenium IDE was recording, pasted the resulting selenium commands into my tests, then converted those commands to equivalent Selenium RC Fu commands. For example:

Your selenium tests will be exercising your application just like standard unit or integration tests. These tests require data and there are a variety of techniques to make data available to your tests. Building your happy path tests is easier if you have fixture data and that fixture data is loaded in your development environment via rake db:fixtures:load. If you use fixture scenarious, it is handy to have a scenario for selenium testing, which you would load with something like rake db:scenario:load SCENARIO=selenium.

With your fixtures loaded in your development environment you can see exactly what is present during testing and the selenium IDE will record exactly what is played back by selenium.

XPath Checker

Usually Selenium IDE can assign a locator that simply works but there are times when it can’t or won’t. Using XPath Checker you can right click on an element and it will display an XPath to that element. You can also experiment with varitions and XPath Checker will list all of the elements on the page that can be identified by that XPath.

Recently I was about to check in some changes and did a last minute click through of the application. All of a sudden I’m staring at a stack trace. My tests were green and I had functional tests for the failing
controller/action.

Tests are like pants — they cover your backside while you focus on other things like adding features to your application. Suddenly I felt a breeze on my cheeks. Something was amiss.

I soon discovered the action and its associated tests had diverged over time. Some of the parameters were renamed in the action but not in the functional test. Since some of the work of the action was
conditional on the presence of certain parameters, that work was no longer being tested.

This exposed weaknesses in the tests and code, such as expected side effects in the tests that are never checked. If they had been checked the tests would have failed and the parameter name mismatch
would have been discovered.

Most functional tests provide specific parameters that should at least be examined during the processing of the action. Reporting unread parameters would strengthen those tests. It was conceivable to me that
some of the other functional tests had similar unused parameters. I wanted all of my functional tests to report all unused parameters.

The first step was to instrument the params hash. I wanted to track access to the params hash and report parameters that were not read during the processing of the action. I don’t know what all is done to params during the life cycle of a test. I’m only interested in access
from the time the action starts till it returns so I need to be able
to turn the tracking on and off at specific times.

It turns out that Rails uses a subclass of Hash called HashWithIndifferentAccess. I added my changes to HashWithIndifferentAccess in test/test_helper.rb:

…is a utility for executing commands in parallel on multiple machines, such as for automating the deployment of applications

Three of us talked about capistrano, deployment and scaling. My talk was titled “Bread and Butter Capistrano” because I wanted to emphasize that I use it as a tool to get my work done, find it very helpful, and use it all the time, but I haven’t spent a lot of time studying the tool itself.

I’ve been using a tool for user interface prototyping and testing called “DENIM“. DENIM is an application that allows you to sketch a user interface using a tablet connected to your pc or mac. It allows you to zoom in and out to create varying levels of detail as needed.

For example, often you will first sketch the general flow of the site. This will look like boxes with a textual name or title. You can then draw lines between these boxes indicating a flow from one box to the other. With DENIM, you can then select a box and zoom in. When you do this the box becomes a page. You can then add details to the page, such as (sketched) forms, buttons and links. Now you can draw a line from one of these sketched buttons to another page and the sketched button becomes a live link. What does this mean? Well, DENIM allows you to save your prototype UI as html pages with clickable images–your sketches embedded in html. When you click on one of the hot areas, which are blue (non hot areas are black ink on white, hot areas are blue ink on white), you jump to the target page.

Suddenly you have a live prototype. I’ve shown these live prototypes to customers and colleagues. At first people seem perplexed. They seem to think I’ve scribbled on a tablet, taken screen a shot and this is how I’m going to show them the proposed UI. Then they click on a link or two and suddenly they get it… and off they go exploring the whole site, spewing out suggested changes. After a few minutes you go back to your office and spend a few hours coming up with a new version. A couple iterations of this and you and the customer have a pretty good understanding of what is to be built.

But It’s Not All Rosy

This program feels abandoned. It has some bugs. The web site has a video of a future version, but the video is a few years old and the features in the video have not been released. I’ve learned to save my work often, and to save it as different versions. Occasionally your DENIM model will become corrupted and sometimes you cannot fix it, so you have to revert to an earlier state. DENIM creates snapshots for you but I lost trust in them early on and haven’t tried again. DENIM also seems to have a size or complexity limit. When the complexity of your UI exceeds some level the model can no longer be opened. I’ve also noticed that some machines can open some models while others cannot — perhaps it is a memory issue.Even with the bugs DENIM is a worthwhile tool to have in your bag. Please check it out.