Pages

"Some birds aren't meant to be caged, their feathers are just too bright"- Morgan Freeman, Shawshank Redemption. This blog is from one such bird who couldn't be caged by organizations who mandate scripted software testing. Pradeep Soundararajan welcomes you to this blog and wishes you a good time here and even otherwise.

Wednesday, July 30, 2008

Most test cases fail

Most testers who write test cases carry a hope that most test cases they write would help in finding most bugs.

Most testers who execute them carry a hope that most test cases they execute would help in finding most bugs.

Most testers who add more test cases carry a hope that they will help in finding most bugs.

Most customers insist on most of the testing to be done from test cases.

Most managers who ask testers to write test cases carry a hope that their testers will catch most bugs with the help of test cases.

Most managers who asked them to write test cases ask a question: How many test cases passed? when considering to ship.

Most testers in those contexts reply, "Most".

"If a test case didn't help in finding a bug then the purpose of that test case failed. Products are shipped when most test cases fail their purpose of existence."

"The test doesn't find the bug. A human finds the bug, and the test plays a role in helping the human find it." --

Update: The comments posted as Angel and Devil are by me ( Pradeep Soundararajan ). Wondering why I posted comments on the post I wrote - you would know the answer if you go through those comments.

The Angels and Devils concept in Testing is inspired by a presentation from Michael Bolton and Jonathan Kohl . Here is an excerpt of the presentation and also illustrates Elizabeth Kohl's design skills.

Oh yes, it might not be about pleasing people but if you don’t find security problems, hackers get pleased of your work ( and the developer who did not fix it if you reported it and he chose not to fix )

Angel said, Oh yes, its not about pleasing people but if you don’t find security problems, hackers get pleased of your work ( and the developer who did not fix it if you reported it and he chose not to fix )

devil wrote This is funny. Writing test cases and getting it reviewed and approved are a part of process we follow to ensure quality.

Oh, you ensure quality. So is quality about those five thousand test cases passing?

If I ask the customers you serve on what quality means to them - would they say - "If 5000 test cases pass then I would call it quality" or would they look at the value the product adds to them for the price they pay?

angel wrote, If I ask the customers you serve on what quality means to them - would they say - "If 5000 test cases pass then I would call it quality" or would they look at the value the product adds to them for the price they pay?

We cover most of things that we think what our customers would call as quality.

For which I would answer: No one can do a complete testing and I am surprised you don't know that.

Hmm! My question was when you guys know "most" you must know what "complete" is to achieve "most".

For instance if I am racing in a Formula 1 car and I have lapped the circuit 49 out of 60 laps I am supposed to complete, I might be in a position to say, "most of my laps are done". If I didn't know what completing the laps means, would I ever be in a position to say "most"?

Do you think customers would give us project and expect to ask such a silly thing?

How silly might the customer think of you for not asking this question and saying, "as 5000 test cases we wrote passed during execution we say it's good quality"?

devil wroteYou must understand that is a team of 5 people are asked to explore then the coverage they achieve would not be known.

I observe a common pattern: All those who talk against scripted testing are the ones who have burned their hands and have witnessed projects getting burned by scripted approach and those who talk against exploratory testing aren't someone who practiced exploratory testing as seriously as it should be.

Moreover I might suggest you to look into things like Session Based Test Management at www.satisfice.com to know how to achieve coverage through exploratory testing.

devil wrote Exploratory testing is about finding bugs by playing with the product or its ad hoc about going here and there based on the experience and finding bugs.

Ah! So its this idea that makes you say whatever you are saying.

How about looking at the idea of exploratory testing from the man who coined it ( Cem Kaner ):

“Exploratory software testing is a style of software testing that emphasizes the personal freedom and responsibility of the individual tester to continually optimize the value of her work by treating test-related learning, test design, test execution, and test result interpretation as mutually supportive activities that run in parallel throughout the project.”

Test cases help in educating new testers who come on board to the project.

I have three questions:

1. Is the mission of testing: To educate testers who join in between the product development OR To find information about the quality of the product?

2. I agree that the education part is important but how about recording testing activity and presenting a video. For instance Test Explorer ( a tool developed by David Gilbert ) records testing activity done on windows based products and throws out a video. How about that?

3. If I were to educate you on how to fly an aeroplane, would you prefer to learn by flying the aeroplane with someone experienced sitting as a co-pilot or by flying with scripts which say,

Step One: Turn on the engineStep Two: Increase the throttleStep Three: Run in the runwayStep Four: Take Off

Thanks Mr. Angel and Mr.Devil, your conversation/debate was very informative and you have talked on lots of topics upon which a "scripted tester" and an "exploratory tester" debate on all the time :-)

This article is a class apart from all other articles which I have read on this site or on other sites. Pradeep you have put in an excellent effort to give direction and explanations to this debate. Your Devil and Angel characters were great. It seemed that you can go on for ever with your advocacy on exploratory testing.

And obviously I got answers to many questions which I had in mind and now I can visualize an Exploratory Teseter in a much clearer way.

Posts & Comments

Search this blog

Loading...

Copyrights

Tester Tested! by Pradeep Soundararajan is licensed under Creative Commons. You must owe credits to Pradeep Soundararajan when you copy paste anything from here by mentioning the name and proper linking to the post. You are not allowed to edit any of the post without permission. For permissions, write to pradeep.srajan@gmail.com