Testing, test management and generic techy stuff

Main menu

Monthly Archives: February 2014

Do you serve your company owners, maximizing short and long term profitability?

Do you serve your manager, fulfilling the tasks she has given you to complete the project?

Do you serve your test team, pulling your weight to get the job done?

Do you serve the developers you work with, making their daily life easier with perfect bug reports and reducing the heat they get from support when bugs slip through?

Do you serve the folks over in the sales department, giving them products that your customers recommend to their friends?

Do you serve the first line support guys, making sure they don’t get flooded with unnecessary support cases and bug reports?

Do you serve the end users, polishing out those little annoyances so they can get on with their work without even noticing the tool they use to do it?

Do you serve the public, pointing out that CPU-hungry procedure that wastes fossil-fueled energy?

Ideally, you are serving all of them at the same time.

Inevitably, there will be conflicts.

To the sales guy, maybe delivering on schedule to the paying customer is more important than giving a completely annoyance-free application to the end users in the customer’s organization. To the developer, those branding issues that the board cares about to build market recognition are just annoying to have to deal with. Digging down to find that CPU hog to save the environment might mean that your team mates have to work harder to get the other tests covered.

The balance is up to you. This is where your professional integrity comes in. Where do you put your efforts, when you have to choose? When are you satisfied with a job well done?

Test cases are a controversial topic in the testing community – for a large part thanks to testing guru James Bach, who has been arguing strongly against the value of test cases.

At the bottom of it, my views are not that different from Mr. Bach’s, but I would still like to moderate the anti-test-case view. I think test cases serve some important purposes.

First off, it is important to underline the difference between a test and a test case. One of Mr. Bach’s pet peeves is that running test cases is not testing, it is checking. I agree – to a point.

My main argument in favor of using test cases is that the human brain is notoriously adept at fooling itself. (Manual) test cases can be an effective method for reducing the effect of availability bias, making sure you cover areas you would otherwise overlook or dismiss as less important. Indeed, the very process of creating test cases can be a way to force yourself to reduce cognitive bias, by forcing your brain to slow down. (On a related note, I strongly recommend for any tester or other person whose profession requires critical thinking to read Daniel Kahneman’s excellent book on cognitive bias, Thinking, Fast and Slow.)

Test cases can also be a good tool for communication. A good test suite can be an excellent way for a fresh tester on the team to learn about the intricacies of an application or non-obvious business requirements of a feature.

Another point in favor of test cases is the sad fact that we live in an imperfect world. While I in principle agree with the idea that anyone who does testing for a living should be awesome, and that a product should never ship without having been tested by these awesome individuals until they are satisfied, the truth is that not every tester is awesome (yet), and not every product release has an available herd of awesome testers, with deep knowledge of both the business and the application in addition to well-developed software testing skills and enough time to apply those skills before deadline.

There are applications and situations where shipping without going through awesome testing by awesome testers is unforgivable. If you are developing an internet banking service, a government self-service platform for tax returns, or the software that controls a medical robot for surgery, there is no excuse for not using top skilled test engineers. But to be fair, the vast majority of software does not fall into this category. Budgets are short and deadlines are shorter. If you have a well-designed test suite created by (or at least reviewed by) good testers, you are better off getting your first line to run those tests, hopefully with the aid of a skilled tester covering the most critical paths, than you would be if you had to ship at the same time and just had the one tester. Even the most amazing testing skill in the world doesn’t always grant you the time to go both broad and deep before the deadline bell rings.

Writing test cases is always a tradeoff, balancing speed, accuracy and maintainability. A test case does not exist in a void – it must be designed with the target audience in mind, knowing how familiar the person running the test will be with the application and particular feature. Sometimes it will be efficient to give just the most brief cues about functions to trigger and things to remember to verify. Other times, the efficient path is to explain step by step how to build up the state that should be checked. The key point for the skilled test designer is to include just enough information for the test runner to know how to check those details that are more important than they may seem.

However, and this is the crucial point: Having a test case in front of you is never an excuse to shut off your brain. Every professional tester should know, and every non-tester who is asked to help out with running test cases should be explained and frequently reminded, that a test case is not a test. The test happens in your brain, as you look at what happens when you perform the steps in the test case, and make a sound, human judgement of whether this is a good result.

Also, though I am an advocate for using test cases, for the reasons explained above, and particularly the one about cognitive bias, it should be clear that I do not in any way recommend only using test cases. As all other approaches to testing, a test suite is a method that should never be used alone – at least not if you want to claim that the product has been well tested.

My first real tech job was for Opera Software. I worked for Opera from 2002 to 2008. As every other Opera employee I already fiercely loved the web when I started. During my six years there I also came to appreciate the intricacies of the web’s building blocks, and the intelligence, skill and sheer passion of those who contributed to create it and make it better. I knew the CSS 2.1 spec like the back of my hand, and had lunch discussions over differing interpretations of the box model.

I had multiple reasons for leaving Opera when I did. I have never regretted it, and even less so after what has happened over the past year or two. But I have often looked back with some nostalgia, dreaming of a world where we actually had specs, where writing a test case was a matter of understanding the specification and converting it to elegance in green and red. I will admit to a moment of looking back with starry eyes when I was reminded of those days, as Opera’s entire W3C compliance test suite, to which I contributed a decent number of tests, was open-sourced recently.

Fortunately, I did not have to stay in Neverland for too long. A stumble across the W3C Memes tumblr site reminded me of the dark sides of the web, of why I gave up on following both the W3C and the WHATWG mailing lists some time after I left Opera. When I did that, I lost touch with most of the community. I also left behind a vast desert of politics, bike sheds and filibustering tactics that would make me scream in frustration. Sometimes I think it’s a miracle that the web has advanced as much as it has over the past ten years.

I still love the web, though. And, knowing what lies behind it, I deeply admire those smart and passionate folks that keep working to make it better, one furiously-debated-over paragraph at a time.

As is customary, every new blog must start with a blog post explaining its existence. This is it.

I already have a blog. It’s in Norwegian, and I’ve used it to write about a range of topics from cooking to politics, through philosophy, literature, and the occasional funny story. What I haven’t used it for much (there are notable exceptions!) is to write about what I do for a living and spend more than the allotted 8 hours a day thinking about.

I’ve been thinking for years that I should blog about these topics – tech, testing, test management – but it never really seemed to fit in my old blog. And so that blog has been all but abandoned for a long time. You know what they say. When you can’t keep up one project, it’s always advisable to start a new project!

Thus, here it is. A blog about testing, about generic techy stuff, about test management, perhaps the occasional office story. We’ll see what it turns into over time.