Review of some testing tools for ruby

by Frank

I never wrote anything about tests on this blog and the reason is simple: I never considered testing to be something fun. The little slogan you see at the top left (a bit cheesy I admit) is the reason why I started Ruby Fleebie back in 2007. I was amazed by the possibilities of the ruby language and the way you could express complex stuff in a very concise an elegant manner.

Now does this mean I don’t write tests just because I think they aren’t fun? Of course not. I do write tests… I just don’t blog about them because they don’t excite me very much. I’m currently on a quest to find a way to enjoy testing more. This is a work in progress for me and so far I have managed to draw at least this conclusion: To enjoy testing a bit more, one need a killer setup and great testing tools to work with.

Continuous testing

Running “rake test” once in a while is a very simple setup. The biggest problem with this approach is that it disconnect you from your “flow”. You have to remember about running this command and if you forget (I know I do) no one will know about it.

So a part of the solution to improve my testing setup was to have my tests run continuously. The most popular library that help with this is probably autotest. It fixes once and for all the problem of forgetting to run my tests.

So we have autotest running our tests in the background. That’s great… but it could be better. the output that Test:Unit or RSpec spits is verbose, hard to read and hidden somewhere on a terminal window. I wanted the testing phase to be integrated as much as possible into my other programming tasks, so I decided to install autotest-notifications, a nice autotest plugin that knows how to talk to libnotify (linux), growl (mac) or snarl (windows) and it tells you if you have failing tests. This is the kind of OS integration I like! I am a Linux guy so I had to install libnotify-bin in my ubuntu box before I could use the plugin. Mac users will have to make sure growl and growlnotify is installed on their machines. As for windows users, they will have to install snarl. Here is a screenshot showing how it looks in Linux :

Other testing tools

Now that my tests run continously, I can concentrate on finding other testing tools that will help me write tests more easily and more rapidly.

Test::Unit / RSpec

I always used Test::Unit. I tried RSpec for the first time very recently and was quite pleased with it. I don’t see that much difference between the two except in the syntax. The only thing I can say is that when I use Test::Unit, I absolutely want to use Shoulda as well. When I use RSpec, shoulda is less essential but I still use it.

Shoulda

I’d say that shoulda is Test::Unit best friend. It gives you a nice set of useful test macros and some great semantic constructs that will improve the readability of your tests.

Capybara (I give to this one the gold medal)

Question: What’s missing if I only use Test::Unit or RSpec ?
Answer: A good integration testing framework

Capybara is a great tool for higher level testing. It’s the kind of testing a non-programmer can understand: when I click this link then I should see this and when I fill this form then it should take me there and so on.

Capybara is a magician that will actually click on your links, fill out your forms and even triggers your javascript. He’s like a end-user testing your app in a hidden browser. Seriously I think these tests are great to have and capybara does a wonderful job to help you write them.

Cucumber

To be frank, I find this one more difficult to love. If there’s one tool I plan to stop using in the near future, it’s probably cucumber. But to each its own!

For those who don’t know, Cucumber is a DSL that encourages Behaviour Driven Development. It lets you describe features before you start implementing them. The “twist” is that you describe these features in plain english, or I should rather say, in plain Gherkin. You then take each “plain Gherkin” instruction and you translate them into ruby code in another file called a “steps definition file”. For example:

Gherkin:

Given a user called Bob
When he stumbles and fall flat on his face
Then he should lose only 1 tooth

Step definition file:

Given /^a user called (.*)$^ do |user_name|
@user = User.create(:name => user_name)
end
When /^he stumbles and fall flat on his face$/ do
@user.stumble_and_fall!
end
Then /^he should lose only (\d+) (tooth|teeth)$/ do |count|
@user.teeth_lost.should == count
end

My problem with Cucumber

I am the co-founder of a startup composed of 2 developers. We obviously don’t need something like the business friendly Gherkin language. For us this language is harder to understand than ruby code can be. Of course, when it’s some dead easy stuff like the example above, it looks cool, but when you start adding fixtures/factories into the mix and that you have several words in the expression that will be replaced in a regex later, then you begin to write features filled with black magic and you won’t want to touch them ever again because you know that a single tiny change will break them. For each Gherkin line I’m writing, I can’t refrain myself from thinking about the step definition that I will have to create for it. That is the main problem I have with Cucumber. As a developer I cannot write Gherkin like it was only plain english. So I often write what we could call technical gherkin easy to translate in a regex-friendly step definition… and sometimes this kind of technical language is a lot harder to appreciate than ruby. Even worse, writing this kind of developer-friendly Gherkin destroys the purpose of this language.

Factory Girl

This one gem will save you a lot of time… once you know how to use it!

What is Factory Girl? It’s simply a way to replace the somewhat cumbersome fixtures in Rails. You can see it this way: Instead of having a bunch of “static” data that will be loaded in your database everytime, with FactoryGirl you build ActiveRecord objects on the fly depending on your testing needs. Basically, you have to define one factory per model. In each factory, you define default values for every attributes you want (generally those that cannot be nil). These values will always be used unless you specify something different in your tests.

If I want to test something with a Game instance and I don’t really care about the other associated models, then I can simply write Factory(:game) and FactoryGirl will take care of creating all the associations for me. I don’t have to create a gameday then associate it to my game… then create a season and associate it to my gameday… then create a league and associate it to my season. I can just type Factory(:game) and I will have a perfectly valid Game instance with all its associations.

I want to bring your attention on line #12 and #20. The way I define my association for the season factory is different. There is probably a better way to do this but I didn’t find any. Basically, instead of going the traditional route and do this:

gd.association :season

I use a block to specify a condition. I do this because of the Game factory defined at line #23. You can see that I have a relation to the Gameday factory and two relations to the Team factory. Both these factories are associated with the Season factory. If I had made a standard association call (i.e. gd.association :season and t.association :season), three different seasons would have been created at line #25, #26 and #27. This is not what I wanted… I wanted the same season to be used.

FactoryGirl is a great fixture replacement, but as you see it got its quirks and it’s not the easiest tool to use.

Conclusion

So, am I a happy tester now? not yet, I’m afraid… but I’m getting there I guess. Feel free to leave me a comment and tell me which tools you use and how you integrate testing in your programming routine.

Bookmark this post :These icons link to social bookmarking sites where readers can share and discover new web pages.

15 Responses to “Review of some testing tools for ruby”

I used to share your thoughts about testing not be fun. Fortunately, now I see a test more like an exercise of “clean code”, and I really like to write code that is as easy to understand as plain english. Right now, I’m using RSpec with Capybara, and creating my own “matchers”.

There is a great book about testing, called “Scripted GUI Testing with Ruby”, and it’s really fun to read. Unfortunately, there are not good resources on the internet about automated tests besides the basic…

Thanks for writing this up! Despite all the emphasis on testing in the Ruby community, there’s remarkably little commentary of this type.

In particular, Cucumber has always been something of a mystery to me. I see the many rhapsodic descriptions, and all I can think is “Why would I ever want to use this?” In fact, this phenomenon has left me suspucious of other testing libraries as well: if Rubyists think Cucumber is terrific, maybe all these other much-loved testing frameworks are equally specious. So it’s nice to see some commentary that is not equally enthusiastic about everything that’s out there.

I have mixed results with running autotest/autospec and use it only when doing unit/model level development. However I’m quite happy with separate continuous integration test running after every commit. At the moment team is small and broken stuff will be fixed quickly.

Agreed about Cucumber. I don’t like it at all and think it’s way, way overhyped. I’m in the process of reading the RSpec book right now and it integrates Cucumber in the whole process, which is frustrating cause I have to learn all this Cucumber-specific stuff just to get through the RSpec examples. That said, the other testing libraries that are given a lot of praise in the community are very good. In particular I’ve enjoyed using Shoulda and Factory Girl.

Concerning the Factory Girl example:
Wouldn’t it be more readable to simply define a “main_season” variable (or whatever you want to call it) to hold a “Factory(:season)” object and simply define your associations (line #12 and #23) to use that value instead of the less intuitive “{Season.first || Factory(:season)}” construct?

@Jimmy Cuadra:
Are you sure to really read the RSpec Book? the one from David Chelimsky? Because last time I read it, there was no traces of any cucumber syntax in the RSpec examples / parts (whereas the contrary is rigth.)

As Erm wrote, Cucumber was designed for one thing: make Acceptance Testing being automatable (for the programmer’s benefit) while still being readable by (non technical) customer.

If you do not have to seat in front of such customer, then Cucumber may be not the tool you want to automate your AT.

Instead you can use Webrat/Capybara/Selenium directly or have a look at Steak (already mentioned by Priit Tamboom in his comment).

IMO, this las tool is only one example of what non-techy people reproach to programmers: being to self-minded with the tools they offer to the good people, especialy when it comes to work with them:
Steak does the same job as Cucumber, except when you “share the code” with the customer.

About cucumber, if there are people who actually find it useful then it’s great. For me this tool is an additional layer that gets in my way. I prefer to use Capybara directly with Test::Unit or RSpec. Thanks Priit Tamboom for refering me to Steak. I heared of this library before and now I think I am ready to give it a try.

@Mauricio, I like the part where you say “I see a test more like an exercise of “clean code””. It’s true that when you take the time to write a good test for something, generally it means that you have (or will) take the time to write good code to go with it.

@GSnyder and Jimmy Cuadra, I’m with you on this! And I think we’re right to be on our guards when the hype is so huge.

@Erm, we also do consulting work so yes we have customers in the pure sense of the word but honestly I think I would never dare to show them a single Gherkin test. They would be so lost! My experience with customers is that they generally don’t want to be involved into the technical aspect of a project. They just want us to understand what they want and not be bothered about anything. I guess it all depends of the type of project you are working on and the type of customer you are working with.

@Frédéric, don’t quote me on this but I think you cannot use instance variables inside a factory because they only exist in the scope of the factory you define them. I’m too lazy to try it right now but I remember trying it before giving up and going with my other (not very good) solution.

My philosophy is that I can write happy or user level cases fairly well, and it’s fairly clear to me what the requirements are. I put more black-box, edge case, and one-off testing in rspec.

I find cucumber is great for flow — I go here, do x, go there, do y, and voila. An end to end test, and it’s fairly clear what users have to go through to do a task. I also find that my test progress with tasks in cucumber is like climbing a cliff to a slide — i spend the first 75% of my time writing the matchers for my first 1-2 scenarios, but after that I get to reuse them and I just fly and crank out a dozen scenarios in a couple of hours.