The SitePoint Forums have moved.

You can now find them here.
This forum is now closed to new posts, but you can browse existing content.
You can find out more information about the move and how to open a new account (if necessary) here.
If you get stuck you can get support by emailing forums@sitepoint.com

If this is your first visit, be sure to
check out the FAQ by clicking the
link above. You may have to register
before you can post: click the register link above to proceed. To start viewing messages,
select the forum that you want to visit from the selection below.

Rails testing semantics are all muddled

I've posted this on the rails mailing list but I thought I'd post it here too.

Yes, I believe it is and I think it is dangerous, and confusing to people new to the concepts of testing.

I've mentioned a point similar to this on here before, but with the announcement of Rails "integration" test, I'm worried that somebody somewhere is getting their types of tests worryingly mixed up.

Lets start with what we had in the beginning...unit tests (for models) and "functional" tests (for controllers). This was wrong from the start, because Rails' idea of functional tests were in fact just UNIT tests for controllers (or specifically, single actions in a controller). A unit test being, a test that tests a small unit functionality at the code level - it makes sure that that unit of code does what it is supposed to do. In a Rails app, a unit might be a single Model or it might be a single action on a controller.

So what is functional testing? Functional testing is another name for ACCEPTANCE testing. What is ACCEPTANCE testing? I will quote from this article:

Acceptance testing, also known as black-box and functional testing, is a way of testing and verifying that an application works according to functional, non-functional, and other important stakeholder requirements. Acceptance tests complement unit and integration tests, which are usually written using xUnit frameworks. Acceptance tests can also be written using a programming language, but Selenium and similar tools like Fitnesse also support tests written using a tool-specific document format.

Further more, there are some important rules regarding acceptance tests:

Acceptance tests differ from unit and integration tests in the following ways:

* The application is tested as a whole, end-to-end entity, not just one class or a set of classes as it is with unit and integration tests.
* Acceptance tests are performed against the user interface using, for example, a browser against a Web application interface.
* The person writing the test cases does not necessarily know about the internals of the application, hence the designation of black-box testing. Non-technical users can write acceptance tests.

The key point here is that acceptance tests make sure that the system works end-to-end and complies with its core requirements. In Extreme Programming terms, this is often a "user story". A user story is considered complete when all acceptance tests pass. Unlike unit tests, which should ideally be running with a 100% pass rate, acceptance tests start at 0% and finish at 100% when the story is fully implemented. From the extremeprogramming.org website:

Acceptance tests are black box system tests. Each acceptance test represents some expected result from the system. Customers are responsible for verifying the correctness of the acceptance tests and reviewing test scores to decide which failed tests are of highest priority. Acceptance tests are also used as regression tests prior to a production release.
A user story is not considered complete until it has passed its acceptance tests. This means that new acceptance tests must be created each iteration or the development team will report zero progress.

So now, we have the announcement of Integration Tests for Rails. Integrations should be tests to ensure that multiple units work together correctly. But Rails integration tests look just like acceptance/functional tests though (much like FIT or Selenium in driven mode)! How confusing for the testing newbie!

The reason I bring this up is because I really believe the Rails developers should look at clarifying and renaming their tests to get the semantics correct. How can you expect testing newbies to start testing their apps when the framework they are using can't even get the names of their tests correct? So what do I propose:

* Rails' existing "functional" tests should be correctly referred to as unit tests, which is all they are. For organisational purposes I can see the advantage of breaking up your model and controller unit tests as the unit testing strategy for these two different types of object are usually different.
* Rails' new "integration" tests are renamed to functional or acceptance tests, which is what they really are. They should be seen as a built-in alternative to using a browser-based acceptance testing tool like Watir or Selenium. In fact, Rails' new built-in Integration tests could be compared to something like Selenium in "driven" mode instead of test-runner mode. Personally, I feel that for web apps Selenium in test-runner mode is more appropriate, but thats just my preference.
* Integration tests are left to the user to put together if they feel they need to use them. Their typical usage would be if there are several models that need to work together to produce results.

To support the above the tests folder would probably look something like this:

I'm going to have to agree with you that refering to acceptance tests as "integration" tests is a bad idea. When you posted to the rails-dev list, what kind of resonses did you get. Any logical reason for doing it this way?

......maybe they are counting on confusing newbies so they can write a book about "testing with rails" to clear up the confusion. ..... *Chaaching $$$*

<quote from imb article>
Driven scripts are also more dependant on the application runtime environment. For example, the Java driver uses an embedded Jetty or Tomcat instance to deploy the application under test. Currently, an effort is in progress to integrate Selenium into Ruby on Rails, but at the time of writing it was not yet released.
</quote>

Are the Integration Test in Rails 1.1 simply an interation of Selenium into Ruby on Rails?

I don't think the newbie is the problem. The Rails framework is my introduction to automated testing, and it seems fairly straight forward.

The problem will be for intemediate developers like me, who might want to take what they've learnt in the Rails environment, on to other programming frameworks only to find that the terminology is different everywhere else. That will be confusing!

I'm sorry to re-live this "thread-corpse", but I'd really like to thank the author for this very un-confusing article!

Since some days I'm trying to **** into testing of Rails applications, and I posted many threads on various forums about the questions "What is unit/functional/integration testing?".
This article here has made many things much clearer to me. I really agree that the Rails developers should over-think their namings for their tests.

Anyway, I have some questions...

Personally, I feel that for web apps Selenium in test-runner mode is more appropriate, but thats just my preference.

Can you explain, why you have this preference? Isn't it very potential to have tests in driven mode that are dynamic? E.g. when changing a route in my Rails app then instead of re-writing every single updated link in my test-runner script the driven script would automatically adapt the new route?

Integration tests are left to the user to put together if they feel they need to use them. Their typical usage would be if there are several models that need to work together to produce results.

Now I'm a bit confused again! Didn't you say that integration and acceptance testing is the same? So why splitting it again in different directories?

Are the Integration Test in Rails 1.1 simply an interation of Selenium into Ruby on Rails?

Definitely not. It's quite similar to the "functional" testing but with some helpers acting slightly differently (see the Agile Rails book 2nd Ed for more on this topic).

And please, tell us what responses you got from the Rails mailing list.