Unpopular Developer 2: Down With Big Testing

Have you ever written tests so big, so comprehensive, that every time you change
something in your code, you have to change your tests accordingly? Have you
ever spent more than 30 minutes writing functional tests for a controller,
including individually submitting nils and invalid data to each action in turn?
How about, have you ever had to use class_eval to awkwardly construct a set of
functional tests that did similar things, because Rails says you can only do one
get/post call in a functional test? This is all nonsense. If you want to
stay sane as a tester, relax and write the tests that you need and not the
tests that your needs-100%-comprehensivity-at-all-times part of your brain
wants, you will find that you are a whole lot more rested, and in general
you’ll bleed a lot less.

This is not about eschewing testing, this is about saying No to Big Testing.
Like Big Oil before it, Big Testing is gaining a lot of clout in the Ruby and
Rails community. Plugins like Rcov and Heckle are emerging as ways to
ensure your code is production-ready, and that in your flawed human state you
have not erred and left out a crucial test. How will you compete with Google
Docs & Spreadsheets if your web-based office application crashes when someone
tries to take the square root of a negative number?

As long as your application has a certain critical mass of stability, stability
has very little to do with whether your idea will take off or not. It’s all the
design choices you’re making, and whether your visual design is appealing and
professional. You know, the meat of your application. The skeleton
underneath should be strong, it should work, but you don’t need to have so much
monitoring equipment hooked up to it that every time you want to shift some
bones around, you first have to untangle knots of electrical cords.

Here’s a code sample to break up this post and make it seem more visual. Don’t
study it too hard, it doesn’t have any relevance:

defselect_birdself.stoked??@bald_eagle:@eagleend

When I was first introduced to formal unit and functional testing in Rails, I
understood it as a way to refactor with ease. If you wrote basic tests, that
made sure all of your features worked as expected, you could confidently go into
a total rewrite of how they worked under the hood. If your tests passed, you
didn’t have to be nervous. This is why I love tests. Little tests. Simple
tests. Tests that touch the outside only, to make sure that the application is
what the tests ordered out of the catalog. If you’re changing tests when you
change code without changing design, those tests are either poorly written,
or ultimately meaningless. For testing at that level, internal or public QA
will take up far less of your time and money.

The three different kinds of testing (unit, functional, and integration)
necessarily attach themselves to your code at three levels of granularity, with
unit testing being the most code-specific, followed by functional, and then
integration testing (which barely cares about your code at all). I understand
that it is impossible to write unit tests and functional tests that don’t
straitjacket the specific implementation of your design to at least some
significant degree. My answer to this, then, is to write far less of them.
Spend less time checking values in the assigns hash, and spend more time on
ensuring realistic uses of the system work properly.

Testing is an asymptotic curve. You can spend 1 hour on a set of basic tests
that cover 80% of your code, or you can spend a large bag of hours on covering
99.9% of your application. Most of the time, it’s not worth it. Write a
functional test for every action that ensures it works as expected, write any
additional interesting, caveat-y functional tests that come to mind without you
really thinking about it (but not if it takes too long!). Don’t worry about
nils or negative numbers, just move on and write a few unit tests to check other
interesting model interactions. Then spend an hour writing a few key
integration tests that really take your application (or application-to-be, if
you’re doing test-driven development) for a spin. Then go back to working on
design mocks for your next feature, or spend time talking to the friends you’ve
asked to try your application out.

If your application has gotten so big and well defined that you don’t expect
there to be any significant refactoring, and the app has become so mission
critical that near-100% stability is a requirement, and you’re ready to lock
down the codebase into a stable, predictable machine, then you should get Rcov
and Heckle and spend an indefinite amount of hours writing Big Tests. But if
your application is at that point, that sounds pretty boring, and I would
find/found another company.