I'm QA with some programming experience, so I am able to debug and review the code of the system under test. I also often read the code to understand how to define request or what response should I expect, if documentation is rudimentary (yea, I know system under test should not be treated as a expected result reference...).

Reporting potential bugs

It happens that during such a review or debugging of some problem found during tests I find another bug. For instance, I find that parameters passed to the client stub are not forwarded in Web service request. From Web service documentation I conclude this may cause integration defect (for instance, it may change the semantics of the requests or cause service failure for a mandatory parameter).

I report this as a bug and describe potential effect, but I have no time to implement full test to provoke such failure. Still, developers keep asking me to specify steps to reproduce the bug and an actual effect and impact.

Is it fair to report a bug discovered during review without performing a test?

Reporting API quality

A subset of problems I find do not impact application behavior but refers to API quality. For instance, I find subsystem API with superfluous parameter that is never used by the component, but confuses both developers and testers using this API ("what value should I put here?").

When you do report a "potential bug" but haven't actually performed a test, what happens when the bug is fixed? Will you test it then? Or just read the code and call it good enough?
–
Joe StrazzereNov 13 '12 at 22:18

Those I found I would re-review, because they were rather trivial and obvious. I want also to focus system tests (that I will perform, when test environment is ready) in that area.
–
dzieciouNov 13 '12 at 22:30

5 Answers
5

I don't think fairness is the issue. More important: Would a report be useful?

For the web service usage, the usefulness of your report depends partly on the scope of your review. Given the code you reviewed, you have a concern. But perhaps other code (outside the scope of your review) prevents the problematic scenarios you're concerned about.

Unfortunately, at this point we don't know whether the problem is a problem, and we won't know until someone investigates further. But everyone who could investigate is attending to other priorities. This is a common issue with bug reports: Who is responsible for investigating? You want the developers to do it; they want you to do it.

What you're learning is that this bug report is not sufficient to inspire others to investigate. At this point I'm inclined to invoke The Responsibility Razor: If I want something to happen, I'm responsible to make it happen. You'll have to either investigate yourself, or do something more to persuade others to investigate.

It's possible that your investigation need not involve running a test. Perhaps walking through the code with some specific values will suffice. Whether this will be persuasive depends on the plausibility of the values you choose, and various people's assumptions about what the web service will do. But I'd start there.

If you don't have time even for that, that's a sign of how important this is for you. So make your report, and let it go.

For the API, the usefulness of a bug report depends partly on the API's audience. If the API is used only internally, and only by a relatively small group of people, I would leave it up to the internal users to decide whether it meets their needs, and to change it if they wish. If the API is used by people outside your team, perhaps you can gather information about their use of the API. If it causes trouble, it's a bug.

All answers are helpful, but this one names the problem the problem precisely "Would a report be useful?" and explains that someone need to find time and investigate whether it really is problem.
–
dzieciouNov 15 '12 at 6:44

I think you should ask the developers and business owners rather than us. :-) Ask them how to handle these kind of border cases. Do they want bug reports also on unclear things to remind them or do they want to talk about them first and decide case by case whether to file a bug? Remember it's your task to provide them information on what is or could be wrong, but in the end it's their decision what to do with the information.

One quite nice way of handling the API quality kind of issues is to have separate category for things that should be improved, but won't be done just now. At least if someone at some point comes back and does something about them. If nothing is done, it's actually worse to have this kind of list than just bug/no bug decision.

Thanks, Edu. In your practice, where you put such a new category for things to improve? In bug tracking system? Or you have some separate system to track technical debt? One options I see could be tools for collaborative code review that let you annotate the code...
–
dzieciouNov 13 '12 at 19:55

All those could work fine. Bug tracking system would be probably easy place, if you can enter them in a way that doesn't bother everyone every day.
–
EduNov 13 '12 at 20:05

SOAP UI is just a part of the game (on test driver part). The system is complex and relies on external components, so setting up environment just for this single test is expensive. Also, since the request is forwarded by many subsystems, I would need to verify it in the very last one. But you're right, I may challenge myself how to setup whole environment for tests faster/easier.
–
dzieciouNov 13 '12 at 19:46

Is it fair? I suppose that depends on the context - who you ask and what their definition of fair is.

I think an important consideration of opening bugs without evidence of a problem is that it could reduce your credibility in the eyes of the bug report readers.

Bug Reports are the primary work product for testers. They are the tangible work most seen by others (programmers, analysts, managers, etc.) and because of this its important, when we write bug reports, to make sure they reflect something we want others seeing. That they are well written and contain the appropriate elements to get the bug fixed. If we write bad reports we loose credibility. If we write a bunch of good reports we might gain credibility.

If you don't have time to test something to prove (evidence) that its a bug (as opposed to an issue, enhancement, etc.) you can always make a note of it and return to it later. If you open it in your bug tracker you might be able to find a way to keep it assigned to yourself until you have time to verify / find evidence of the problem.

If you have questions about an aspect of quality (is this a bug?) but are unsure you can ask in a more informal way (assuming opening in a bug tracker is formal) - in person, email, instant message, whatever. When you have evidence that it is a bug is when I generally recommend writing it up formally. Then again this all depends on the organization.