This blog is about thinking of things past, present and future in testing. As much as I'd like to see clearly, my crystal ball is quite dim. Learning is essential and this is my tool for that.
A sister blog in Finnish: http://testauskirja.blogspot.com

Tuesday, April 26, 2016

How would you count bug reports you never get?

There's recently been a significant transformation on a piece of software we work on with my team. In the last month, we've done more releases of that piece than we managed to do in the year before it. There's a clear feeling of progress. And the progress has included the amount of code for the piece dropping to a quarter of where we started off with the transformation. The final frontier of siloed development has joined team ownership and our drive to clean up things to be understood.

Looking at this piece of software, I'm observing an interesting aura of mysticism on how users approach it. It's like I'm moving into the past where I four years ago surprised our product manager with the idea that software can work when he tries it out - something that should be close to obvious in my books.

The worst piece of code my team delivered probably had also just one or two bugs in production over the last year. But for us, the reason was that the users would seek workarounds rather than ask for fixes, or that the bugs just did not bug the users enough for them to speak up.

Since we moved the piece of software into the frequent release cycles, we've fixed a LOT of bugs users never complained about. A lot that users had not even run into. Looking at the bugs we've fixed, I find it nearly impossible to believe that they could have been using the software all this time. But they did, they found their ways around everything that bugs us (and slows them down).

When you get little to none in bug reports from production, do you assume all is well? My sample says I shouldn't. Even asking the users directly might not give you a realistic picture.

And on the other areas where feedback is more frequent: if it bugs a user, it's a bug. And we have loads of things that bug the user, that we rather frame as feature requests, some of which we have no intention of acting upon. Counting them seems more of counterproductive. We could count how often we get interrupted with a quick request to change something in production that reorders our immediate priorities? But most of those wouldn't count as bugs in any traditional sense.

Yes, my team does both of those mechanisms - now also for the piece of software that puzzles me on how bad it can be.

I'm just noticing that I get more bugs out of every software (with real and imaginary user collaboration) than companies who keep thing they have no bugs in production. Lack of exploratory tester makes me suspicious of how critically the product has been looked at.

This is something I struggle with in discussions with my developers around bugs, to fix or not to fix.

We are always behind the ball in terms of work, so new functionality is always prioritised over bug fixing unless the bugs block the new functionality, are a critical bug (crashing, affects financial data etc) or a user has complained about it.

This however leads to a great many "minor" bugs that simply stagnate at the bottom of the backlog. The reason being, that if they're important they'll be raised by the users. If not, they obviously don't care.

But this argument doesn't work when we know that the act of complaining/sending in feedback is more trouble to some people than them just finding a workaround, or tolerating the bug. But if we have no visibility of the above cases, I can't feed them into my bug advocacy... and the bugs will continue to be "at the bottom of the list".

I've struggled with the same. I found a few strategies that have helped me. 1) I've learned to fix some bugs myself, in particular the types that take longer to report than to fix. 2) I've cultivated relationship with one (or more) devs who will pair up with me to fix stuff that they wouldn't fix by themselves3) I've brought in "real users" to be listened to.