I personally think unit testing is a waste of time. I was fortunate enough to work on a large project which had a vast amount of unit testing in it. I noticed after a year working there that we spent more time on the unit tests than on the code itself, and in all the time I worked there, the unit tests only found 3 bugs ahead of production. After roughly doubling development time (conservative guess; I'd put it at triple).

IMHO unit testing is papering over the cracks in design and specification. Wherever unit testing is being used today would be better served by, for example, design by contract.

But before I make any kind of comment, I want to clear something up. By unit test you mean only the test of something small (like a single method)? Or do you also include functional tests in your war plan? (a test representing for example an entire use case which touches large chunks of your code to test if it works as expected given certain parameters).

I have to ask because the definition of unit test is a bit vague nowadays. Many people don't actually refer to testing something small, but only to a test that can be invoked through junit or testng (I have to admit, I make that same mistake).

'twas JUnit (and using JMock or something for a few of them), on a plain old Java desktop application.

@gimbal - I mean unit testing as is traditionally defined - a unit being one software component, which in Java, tends to be a single class*. This leads to designs that aid unit testing, which in turn leads to designs that I would not ordinarily design, which again, I'm not sure is necessarily a good thing. Though of course proponents of unit testing claim that designing classes just so they can be tested with unit tests is a Good Thing.

As soon as you're getting into the testing of the interactions between two or more classes you're into system testing territory and I've seen plenty of people abusing unit tests which do exactly that, and is yet another reason why unit testing usually seems to fail to do quite what it was supposed to do.

And of course there is frequently the observation that testing components completely in isolation does not seem to actually find many bugs. I merely observed this in my own experiences of the project I was working on. Before I get too smug about that though it would be wise to consider that the reason the unit tests never found any bugs is because they'd been in place for so long. Perhaps.

Cas

* Though really a single file is probably more appropriate, as classes frequently contain inner classes to help them along.

What language was this in btw? And does your opinion cover all usages of unit tests, or just with that particular language?

Also, were these unit tests, or functional tests but written in a unit testing framework like junit?

I've never used unit testing in any other language so I couldn't comment but I suspect the reasoning and outcomes are the same. And they were supposed to be unit tests but many of them were largely irrelevant or accidentally testing more than one public class by design.

Since this is about unit tests in their intended form, I can only agree with you. Unit testing small things tend to cause an insane amount of them to be created in any half-decent application which eventually only become a maintenance nightmare. Nice idea with good intentions, but in my opinion not really "realistic" in the long run. I especially dislike ruling that exists around it, like "if you fix a bug you MUST prove it with a unit test". Such black and white rules tend to lead to utterly useless tests...

I much prefer to stick to the larger functional tests; in fact they are the tool I use to actually develop most (enterprise) applications; all from the tests (which tend to be in the form of one "happy flow" and a great number of tests that deal with failure paths). I tend to have 90% of the code properly tested before I even deploy a single thing, the remaining 10% is usually configuration and environment related (like FTP transfers not being handled properly, etc. etc.) or something as minor as a mistake in a web resource, and not so often for example an architectural mistake.

The real benefit comes when RFCs come into play that actually alter architecture. After modifying the application I end up with a number of tests that fail because their test conditions no longer apply; usually only tests from the same test suite by the way. Through adapting them to the changes I have a far greater insurance that I actually did it right than if I didn't have the tests to inform me of my misgivings (and perhaps even design flaws; when it is difficult to get something into a test I treat that as something suspect). Plus I have again the benefit of seeing it all happen by simply running a specific test through testng and seeing what happens under the set conditions, either through logging generated or by stepping through. I wouldn't have to for example deploy, go through a web interface and follow a very specific routine to get things to happen, the test does it as a sort of prebuilt script.

I don't diss unit tests entirely, I just don't use them for what the programming gods intended. In for example graphically oriented applications; be it a game or for example a Swing application, I tend to use them to quickly try out code before I start to see them work when firing up the application/game itself. Just to check the numbers and make sure I didn't screw up on calculations and such. Call them disposable unit tests - saves plenty of time.

Humm...I'm old skool. I tend to use this antiquated notion once called design. Then implement and test hedge cases and stick a fork in it.

Seriously, this is one of those notions than have gone way out of control. It should be pretty obvious when some white or black box tested is needed or not. I tend to think that some modern CS notions all stem from assuming that most programmers are monkeys that couldn't get work attempt to recreate the works of W.S. The problem is that following these kinds of practices tend to create monkeys rather than computer scientist...so it a self fulfilling kinda thing.

Its certainly an interesting opinion. Me personally I don't see how unit tests would be used as a replacement for design (if I understand it properly). *scratches ear*. Like anything they're a tool which you can use for good or for bad, the birth of the concept and the overly excited adoption (which was pretty similar to how XML was abused) didn't change anything IMO. Ook, eeeeeep!

@gimbal: With a proper understanding of the problem and a design spec of sub-system, it should be obvious where (and if) given types of testing are needed or not. Too much focus is placed on proper "software engineering techniques" (monkey programming) and too little on problem solving, which IMHO is really what computer science is all about.

When I write some complex method or class, I like to write a small test that verifies some normal cases and some corner cases. This gets rid of most off-by-one bugs. I never unit-test highlevel constructs, as the number of potential (conflicting) parameters spiral out of control: it's indeed a waste of time. That doesn't mean that it isn't a valuable tool to quickly check to see whether a method fails to do what you think it should do, as it might also reveal that your understanding of the method is wrong. Whatever it is, this trivial testing will either find obvious bugs or inform you on how to use it.

You might also argue that unittests are a basic form of documentation.

Hi, appreciate more people! Σ ♥ = ¾Learn how to award medals... and work your way up the social rankings!

@gimbal: With a proper understanding of the problem and a design spec of sub-system, it should be obvious where (and if) given types of testing are needed or not. Too much focus is placed on proper "software engineering techniques" (monkey programming) and too little on problem solving, which IMHO is really what computer science is all about.

Agreed. I think this overlaps a bit with the "design pattern" topic of the general rant thread

Quote

You might also argue that unittests are a basic form of documentation. persecutioncomplex

Unit tests... no, too small. A javadoc suffices more. But the larger tests I was yapping about certainly are a form of documentation - for me; after a year I forget the details of the code, but the automated tests always get me back on track quickly.

A unit test is just more code, not really an explanation of what you're trying to achieve. It looks to me as if unit testing is being overzealously applied to make up for a lack of take-up from design-by-contract techniques. And failing. Everywhere I see it being used in a corporate environment, it's all about ass-covering, "not breaking the build" (haha as if), following procedure ("we do it this way here"), etc. rather than any intelligent application of unit testing as per Riven's usage - that is, just testing a few little odd bits here and there that are complicated.

The most important test is looking at the sourcecode. No joke, this is seriously undervalued.

Anyway. I think it the testmethod depends heavily on how the development is structured.The less a programmer has influence and responsibility for the complete product, the more paralel testing is needed.For the programmer is working under a different contract (of involvement) than a developer in a 2 person team.

Ive seen code that could never have run. Wich means that the programmer never ever called the method before in a testenvironment.More like coding in a style "if eclipse does not show red lines, its fine to check it in, the tester will give feedback if it does not work"NOTHING should ever leave the dev- computer without beeing run at least once in a standard case.

Tesing a component by itself is important when integrating this component adds a huge complexity to it, such that initiating and tracing bugs is more timewasting thanwriting a test-driver (and testcases to check borderconditions) for it.

Most bugs arise either in novel aproches to solve a problem (doing something methodically wrong)or during maintainance when repetatively extending parts to cover additional requirements - using other parts as a template(copy pasting and adjusting other codeparts -> there is the danger of ASCII degeneration, where copying Text might loose single bits)

design-by-contract is awesome. It's a shame that I've never liked any languages that fully incorporate the notion, but that's neither here-nor-there.

As an additional aside, the JSR that deals with allowing more type annotations have a number of examples that 'hint' that there may be some type contracts headed our way, such as @NonNull, @Nullable and @Immutable.

A unit test is just more code, not really an explanation of what you're trying to achieve. It looks to me as if unit testing is being overzealously applied to make up for a lack of take-up from design-by-contract techniques. And failing. Everywhere I see it being used in a corporate environment, it's all about ass-covering, "not breaking the build" (haha as if), following procedure ("we do it this way here"), etc. rather than any intelligent application of unit testing as per Riven's usage - that is, just testing a few little odd bits here and there that are complicated.

If you seen this applied the way you described here, it wasn't right. Unit testing is mostly used in TDD (test driven developpement) and it has been demonstrated that it can cut down the total developpement time of rather big project by factor 2. On corporate project, assuming that every programer isn't a A+ class, it helps to avoid 4000 lines methods and enforce decoupling of the different parts of the application since it force the application to be testable, it's a mean to obtain quality, not just "ass-covering".

My empirical observations were that it increased coding time by at least a factor of two. Every line of code is a line of code that needs maintaining; the unit tests were as big as the code they were testing.

My key observation was the number of actual bugs the unit tests found. That is, about 3, in a year. The buglist itself though stretched for pages and pages. This leads me to think that all the real work of testing working software lies elsewhere.

You see all kinds of very questionable studies...or at the bare minimum don't match anything in my experience. An example is the stupid "programmer pair" thing. I find it very hard to believe that it's a win.

Anyone who makes a programming language compiler/parser: tests cases are absolutely critical. They aren't just some checkbox item. They are critical in supporting the thousands of different types of code that you're expected to handle. Every major compiler has a large bank of tests cases to vet out every release. Web browsers are a variant of this.

Also, data structures. At various points, I've needed to design custom internal data structures for a given application. The test cases were very useful first in writing the data structure and testing out edge cases, but the bigger benefit was in stopping other developers from breaking things.

Certain code doesn't mesh will with unit tests: user interface code is a good example. All the code needed to wire up interfaces and handle user input and update screen elements doesn't lend itself to meaningful test cases.

Language front-ends, back-ends and everything in the middle is a very different kind of animal...and yeah, in this case you need LOTS of test unit like constructs. But that's a whole different animal. It's pretty rare to have sets of sub-systems with this kind of inter/intra dependance.

Also, data structures. At various points, I've needed to design custom internal data structures for a given application. The test cases were very useful first in writing the data structure and testing out edge cases, but the bigger benefit was in stopping other developers from breaking things.

Certain code doesn't mesh will with unit tests: user interface code is a good example. All the code needed to wire up interfaces and handle user input and update screen elements doesn't lend itself to meaningful test cases.

This is my experience with unit testing. Especialy, stopping other developers from breaking things is the point of it. Since the costs involved for the life of an application is mostly about maintenance and not it's initial developpement, it's imperative that developpers write code that test code even at the cost of having to write twice as much code .

IMHO unit testing is papering over the cracks in design and specification.

Absolutely true: unit tests are neither a rigorous nor complete specification, but if you think your program suffices for that, you go ahead and collect your Turing award for writing a better theorem prover than anything else out there. Unit tests are about recognizing that your specification is always going to have those cracks, even if you write it in something like Haskell, which has phenomenal power to specify things statically. Hell, even Agda code often has unit tests, and its type system makes Haskell look like PHP.

It's true though that the more loosey-goosey the language, the more the community considers unit tests to be holy writ. Every time I see python and ruby programs checking the type of parameters, I shake my head and ask why they so strongly resist having a compiler make those kinds of tests for them.

"if you fix a bug you MUST prove it with a unit test". Such black and white rules tend to lead to utterly useless tests...

That's not a unit test so much as it is a regression test, which in my experience is half the time a functional test by nature (testing multiple things) and the other half some kind of corner case that I neglected in a unit test (that tests one thing -- that is, a "unit").

Opinions differ on unit tests, but I can't conceive of any credible opposition to regression tests on known bugs.

I've extensively used JBehave (http://jbehave.org/) and indeed Behavior Driven Developpement is realy something to consider in library writting, I've got nice results with Web Service Interface testing too. The nice thing about this is that anyone can write a new test case can do it, even a non-programmer!

BDD to me seems like extra annotations on unit tests. Might be useful for a QA guy writing the test skeletons that a dev fills out, but it doesn't strike me as being particularly different from unit tests. Something that actually is different is QuickCheck, where the test system generates tests for you based on logical properties you assert about your code. It's still in its infancy for languages like Java, which can't express meaningful properties in the type system, but DBC-like annotations also work.

I'd say it complementary because it's more about integration tests than unit testing. It's very usefull for an analyst, who most of the time don't write code but want to test the application, it's often faster for him to write a test in plain english than retesting all cases one by one by point and click the interface each time developpers make changes.

Yah, I guess I'm colored by writing Specs2/ScalaCheck unit tests, which have a sort of BDD-styled structure to them but are otherwise still plain old unit tests. Frankly QA has no business writing unit tests, but they do come up with imaginative ways to break whole systems that demand a way for them to communicate the specific tests back to devs. I'm not sure jbehave is it, but it beats vague handwavey "write tests in a spreadsheet" nonsense at any rate.

I don't like writing tests.But what I even more dislike is haunting bugs longs after you think you've finished something.

Writing tests after the puppets started to dance is no fun.We should look at tests as being a part of the implementation process. Tests offer a good, fast and isolated sandbox for new functionality, instead of launching the whole application over and over again.

I managed to write at least about 200 tests for my little Indie game which is not much considered the whole code base. But I am really glad to have them.Relaxes the relationship between you and your codebase.

There should be a new, cooler name for tests that could also be admired by the "testing-is-for-wimps-party".

I haven't ever used them extensively, but generally I would agree with you.

However, if I were my friend who works on the computers for an airborne windmill (that will crash if has a bug), or if you're writing the code for a nuclear missile, or anything like that... unit test the shit out of it.

java-gaming.org is not responsible for the content posted by its members, including references to external websites,
and other references that may or may not have a relation with our primarily
gaming and game production oriented community.
inquiries and complaints can be sent via email to the info‑account of the
company managing the website of java‑gaming.org