When correcting bugs, it is encouraged where I work to first write a test that fails with the given bug, and then to fix the code until the test passes.
This follows TDD practices, and is supposed to be good practice, but I noticed it tends to produce cryptic tests that come really close to the implementation.

For instance, we had a problem when a job was sent, reached a certain state, was aborted, and retried. To reproduce this bug, a massive test was written with thread synchronization in it, lots of mocking and stuff... It did the job, but now that I am refactoring the code, I find it very tempting to just remove this mammoth, since it would really require a lot of work (again) to fit the new design. And it's just testing one small feature in a single specific case.

Hence my question : how do you test for bugs that are tricky to reproduce ? How do you avoid creating things that test the implementation, and hurt refactoring and readability ?

Is that error case relevant to the new design? To me part of a new design would be to increase reliability of the design, so you might be able use that to justify removing this test case if this type of bug was anything to do with a mistake in the original design.
–
ThymineMay 10 '12 at 17:03

the refactor is about something else, and it is still possible in the new design to slightly modify the code and reintroduce the bug, which is what the test is watching. I guess an alternative to the test would be a comment in the code saying "don't fuck with this", but it sounds like the wrong thing to do :p
–
ZonkoMay 10 '12 at 17:24

if it's too complex for for a unittest make it part of the integration test
–
ratchet freakMay 10 '12 at 18:59

4 Answers
4

Yes, in general you should. As with all guidelines, you'll need to use your best judgement when they run against other guidelines. For this scenario, the severity of the bug needs to be weighed against the work needed to implement the test and the quality of that test in being targeted to the business problem and catching regression of the bug's state.

I would tend to favor writing tests over not, since interruptions to run down bugs tend to have more overhead than simply developing and maintaining a unit test.

I would add more emphasis to this and state that in an ideal world it would only be considered a bug if a failing unit test exists, but +1 for the italics and noting that business needs should prevail.
–
Joshua DrakeMay 10 '12 at 17:23

2

well, indeed, in the end it's all about a balance between the time needed to maintain the test vs the time it would take for a noob to detect and correct the bug if it happened again.
–
ZonkoMay 10 '12 at 17:31

I would also favor generalizing the test so that it's not just testing aborting and retrying the job when it reaches one specific state, but rather tests aborting and retrying at every state a job could be in.
–
Old ProMay 10 '12 at 18:57

+1 for "since interruptions to run down bugs tend to have more overhead than simply developing and maintaining a unit test."
–
Peter K.May 10 '12 at 19:11

I think the best practice - one I'm embarrassed to admit I don't often follow - is to create a system or integration test that demonstrates the problem observed in production, then research to find the unit(s) responsible for the problem, and then write unit tests for those units that demonstrate the problem at the unit level. Once you have the unit tests, fix the units, and run all the tests. At this point, it may be prudent to discard the original test - because it may be fragile and/or slow - but keep the unit tests in your automated suite for the sake of regression and coverage.

The practice of writing a test to identify the defect is a good idea, since it allows you to identify exactly what steps are needed to reproduce the defect and verify that it has been fixed. In addition, these tests can be run as part of smoke testing or regression testing to ensure that later changes have not re-introduced an old defect into the system.

The first thing to consider is the level of the test needed. Perhaps the test to verify the fix would be more appropriate at a systems level, or maybe even an acceptance test that's done manually. I think it's more important to have a test that's documented and managed, regardless of how it's specifically implemented.

As far as how refactoring affects tests, it depends on the specific characteristics. In some cases, refactoring (or any type of work, such as new features) could make the tests no longer necessary. The problem, as it originally occurred, might no longer be possible. In that case, it might be wise to remove the test from the possible tests to make your test process (automated or manual) more lean. In other cases, there are multiple methods of performing the test and verifying the feature at a different level might be more appropriate. If the feature is minor, perhaps the test is no longer necessary at all.

You can also consider not only relying on testing, but logging as well. For example, capturing information at run-time (with varying levels of verbosity depending on the environment - more verbose during testing, less verbose during deployment), profiling the application, capturing dumps of the current state of the system. If you can find common triggers to the problem, you can use that to guide testing at all levels.

Write unit tests for existing code base. When fixing a bug, you need to make sure that your unit test fails - this will give you confidence that you are indeed working on a bug. You then need to re-factor and make the test pass by fixing the bug.

This is not a TDD practice though. In TDD tests drive your design, but in your case the design has already been decided upon.