Pages

"Some birds aren't meant to be caged, their feathers are just too bright"- Morgan Freeman, Shawshank Redemption. This blog is from one such bird who couldn't be caged by organizations who mandate scripted software testing. Pradeep Soundararajan welcomes you to this blog and wishes you a good time here and even otherwise.

Tuesday, June 08, 2010

Experience report of testing versus checking

Many thanks to my client in Bangalore who encouraged me to blog about the experience, value and outcomes we had in separating tests and checks. If you are reading this, it means my client has approved this for publishing.

Thanks to Michael Bolton who posted a series of posts on his blog under Testing versus Checking and those who commented on the posts, whose ideas and thoughts helped me to think about delivering the value of tests and checks to my clients. Tests and checks series of posts has helped me communicate things better and organize my thoughts better. We were using the term "check" and "test" much before Michael blogged about it; however, he gave a cognitive structure to it in our brains.

The context

One fine evening, there came a client looking for me and the story goes like this...

I was hired for a couple of weeks, to consult and test various products produced by a large organization whose headquarters is in Europe. The moment I was hired, there was a need of a hand to run through a few tests on a product. The product was used for analysis of somethingacross many country deployments. It has an engine loaded with business rules, lots of history data in a database and report viewer application. How do we know if the business rule engine didn't violate any rule? The reports had to comply with complex business rules that at the first sight appeared to me as contradicting one another. The core development of the product was being done by a vendor in the US of A, who was reputed within my client's office for pumping in poor builds, as frequent as they could. I wish I could tell you more than that.

Video test cases instead of lame text

On starting off, I realized that the learning curve on the project appeared to be steep and I couldn't cope with it in the time I had. A demo was given to me from a colleague who had been running tests ever since he was hired. Does a demo suffice? No way.

Building a little credibility with that colleague helped me convince him to record videos of the demo and tests with the help of Windows Media Encoder 9.0 for Windows XP. So rather than following dumb scripted tests, I was watching a video on one monitor and running tests on the other. Video test cases were more reliable than documented ones. Based on how quickly I picked up things, the videos are now being planned as training material instead of asking someone to go through test cases.

Recognizing checks being called as tests

On running them, I realized, what I had been running were checks than tests. To quote Michael Bolton, Checks are machine dependable and tests require sapience. I was checking if the report complied with all business rules. While I was checking, I instantly knew that these checks could be automated.

Testability, Test Coverage Issues & Stakeholder Interests

I noticed a couple of bugs in the software were preventing me to run other kinds of tests as fast as I thought I could. Anything that blocks me or slows down my testing is a serious issue for me and to those who have hired me. With the proposal to test for various quality criteria being accepted, I made a presentation to the stakeholders of how the quality can be improved at least with respect to testability and other quality criteria (such as Usability, Performance, Security.. ) accompanied by a test report.

They instantly liked the improvement of testability idea because to check if the business rules are not violated for every installation, they had been taking 2 - 3 days. They needed rapid feedback to take better decisions.

Imagine looking for a specific value across many excel sheets. By the time you are on the fifth, you might almost forget what you saw on the third and where you saw it. However other kinds of quality criteria didn't matter to the stakeholders for their current context. I wish they had considered it but that's a business decision. My bug advocacy was strong though. I had stakeholders wanting to automate the checking part of our testing. That was a winner.

Partnering with developers

I was sitting in an organization whose policies and people are new to me. I was proposing things that required lot of dependencies on other people in functions such as development, business analysts and on-site personnel with of course the management. The time frame I required to achieve this was really short. Writing emails in a manner that helped the people realize the importance of the value was a big winner for me. For instance, here is an email I sent to the development team that sits in my client's location who mostly do some integration and writing wrappers.

Dear Developers,

Greetings!

I am glad to be interacting with you. As we are committed to reduce the time we take to analyze a release of XXX XXX, we realize that a tremendous saving to the organization can be contributed by making a few changes. We have made a few things from our end, such as, having a formula do the job for learning about XXX-XXX rule being satisfied for all installations. Creating a repository of possible combination of tests we need to do to be able to say we have exercised the business rules very well.

In that path, we realize that with your help, we can do a lot more. I understand that you might be busy with other things on the table but as I feel we are working together towards a common goal, it makes me request you to help us.

We request the following things from you:

Currently we are spending about 3 – 4 hours manually to verify the XXXXXX business case for a small installation.

We have investigated and conjecture that if you help us have the XXXXXX( such as XXX or XXX details ) to be available in a list associated with corresponding XXXXX and the XXXXX, it would take us less than 5 minutes for us to perform the business rule validation.

If these details are a part of the export of Weekly Report (highly preferable) or as a separate excel list (less preferable) it would help us achieve the goal.

Assuming a set of testers will be testing XXXXXX for all country implementation in future, you would be helping us bring down the cost of testing by a couple of thousand dollars ( Well, Euros, as we are European based )

Additionally, as we are doing it manually, we are not exceeding more than 10 categories per installation. With your help we would have the capability to go as much as the number of categories the system can support. Thereby, we test for larger samples and increase our data coverage.

We would be in a position to provide you any help you might need from us in achieving this task or would be glad to offer back any other help that you may require to make your work more effective.

Thanks & Regards,

-- Pradeep Soundararajan

30 minutes, no response to the email. At the 31st minute, a developer was at our desk. It happens that he was serving his last day for the organization and carved out time for us. I considered that as a great gesture by a developer to support test team and our test team brought a chocolate for the help he offered. I couldn't get a "Thank you" card; else I would have wished to have given that as well.

That developer helped us with stored procedures and SQL queries that we need to extract data out of the db.

The Checker Tool Development & Testing

So, instead of looking through reports exported through the GUI, we now had the option to extract values directly from the database. We got one step closer, to what we wanted to achieve.

While I was planning to write a Perl script to extract the values and dump it to an excel sheet, I became aware of a developer in test who joined the organization and had not yet been assigned to a project. The good news for me was she hadn't got her computer to start working. I requested the manager to get her time for helping us write a script and gave her my PC for 2 days while I mostly roamed around saying "Hi" to people around and getting a hang of the culture and talking to people about their work. I could have done all by myself using Perl but I know how slow I'd be as compared to a developer in test.

I provided her with requirements and all necessary tools that she'd need to develop and test the script that could dump the values in an excel sheet in a format we wanted. It's a complex data structure and in order to perform an analysis we had to develop a report format. With the help of my colleague, we zeroed in on the report format and in 2 days time, she could help us see what we wanted in the report format.

TDD for writing Excel Formulas

Off to validating business rules through Formulas in Excel. I knew the power of Excel and its formulas but I didn't know its limitations. I remembered that I had received an email long back from someone containing MS Excel formula dictionary as an attachment. I pulled that out to help myself.

Excel shouted a warning and error if we exceeded 7 nested loops. It didn't warn us when we had a typo in one of the formula, a star * symbol instead of 8. Copy pasting formulas from one sheet to another had reference to the previous sheet. I know lot more about excel than what I knew earlier.

I wasn't doing a good job and I knew it very well. A month and a half back I attended a demo on TDD from Venkat Subramaniam. It struck me at the right time. Instead of continuing to do a mediocre job, I started collecting data, scenarios and information that should make the formula fail or pass. I started to write the excel formulas based on the data we had. My colleague who had been in this project for a while helped me get the set of data that had to result in pass and those that should result in fail. If you may permit me to say that writing formulas as, "coding" then we were pair coding.

This was very helpful. After starting with that approach, we could find bugs in previously written code. Prior to the TDD and paid coding approach, I wrote a formula, loaded the data and checked to see if it failed or passed. For a condition in which it should pass, if it failed, I used to tweak the formula to see a Pass. That was dumb, especially, after I knew that making a Pass could impact other things that were supposed to fail.

With TDD in, we achieved faster progress on our business validation formulas. We were now at a speed lane. We cracked the puzzle too. We could check for business rules violation in a matter of few seconds.

Comparing humans and tools for speed

I am against the idea of comparing human testers with test automation. I still am against the idea but I saw myself in a situation where I was comparing automation with manual effort. Why did I do this? Should I remain silent about what I have done and continue telling people who attend my workshop that; comparing a human and a tool for speed is a bad idea? I was reminded of James Bach's blog post "Manual tests cannot be automated" and Jonathan Kohl'sinterview where they put a lot of things so well that I couldn't have put it then.

In self retrospection, a learning emerged. We didn't automate tests, we automated checks. As humans who were not supposed to be doing checks were doing it, I was forced to make that comparison. In fact, I should have said, "Please do not see this as a faster way to do things by comparing how testers were doing it earlier. They weren't even supposed to be doing that. I just wish there was more freedom and a little bit more authority for them to express it". Getting our tool tested by domain experts

"Yipee! It works". That's not what we thought. When we developed this tool, we were sure that the users of this tool could be beyond the test team and how good our tool was to be decided by a "domain expert". When we showed some of our unit test results to a business analyst in house, he was excited at how simple this would be for him to use when he goes to assess a country installation in the future. He vowed to support all that we needed. He offered help writing us a few formulas that we needed. I tweaked it to suit our needs.

He had a logic that I wouldn't have touched upon. So, that really helped. We didn't need to be a domain expert all by ourselves to be able to do this. A testing brain and a domain expert brain coupled together yielded a lot of value.

Added to that, he was a key stakeholder we had in house. He acted like a customer in house. He suggested the changes we needed to make it more suitable to other stakeholders' usage. Having been on site, he could bring the perspectives of people who could use this tool and their contexts. We needed lot of clarification on the business rules. Imagine a set of rules contradicting each other but still plausible under certain contexts which had to be shown as pass and not fail. Whoof! I liked it :)

Live Testing of Business Rules Validations Checker Tool V 1.0

So, our in house customer was happy with it. We showed a demo to the test management and they were excited. The test manager was excited enough that he pulled out the product manager out of his busy schedule to show our tool to him. The product manager was about to be in a meeting and he asked us, "How long this would take?", because he knew the business rules was taking a few days. When he asked us that question, we initiated the script, took a deep breath and said, "It's over". We showed him the report and there was a smile that our test manager was hoping to see. The only skepticism he had is about business analyst's approval of this tool. A bug in this tool would really misinform the decision makers.

So, the tool was shipped to people in Europe who tested it on the live system. They took about 2 days to get back. It wasn't a very nervous moment at all for me. I was just impatient. They had to take two days to approve this tool because they had to compare the results of the tool against their manual checks. While we were waiting for the reply, we continued testing our tool to see if we could spot problems. We ran it for several test installations and we made notes on how we could improve but no bugs found.

The climax

On the next day when I was traveling to my client's location by bus, I get a call from a person who heads the testing for the group which I was hired. "Are you in the bus? Can you get down? I want to talk to you. I will come there soon". Before I could ask anything he disconnected the call.

That's really crazy. I suspected something must have gone wrong. I thought the Head of Testing wants to give me a bad news that my consulting contracted is terminated. He arrived at the place, I got in, and he showed me an email and said, "Congratulations man. This is great news". The tool was approved by the business analysts who were on site in country installations. They said it was accurate and also proposed some modifications to a dashboard we had created in the excel. It hit the target on the bulls eye.

The Head of Testing and I together prepared a presentation of the whole story to circulate it throughout the organization. The Head of Testing currently hopes the concept of separating tests and checks is carried over across all other projects.

At one side I was extremely happy for being such a value to my client and on the other hand, I was telling myself, "Consultants like me are made to look good for doing what everyone else is supposed to be doing". Had my colleagues at the client location been people who were active in learning, I wouldn't have sounded like an expert tester to them. I just feel pity and can offer the help I can. I have to move on and learn more things.

Preventing bad builds from sneaking in

The story doesn't end there. I had another proposal up my sleeve. I wanted to ship this tool to the vendor in US who was pumping in bad builds. We are planning to use the Business Rules Validation Checker Tool as acceptance checking of a build into the client's location. The idea of automated acceptance checking appears to be interesting to my client since they work with several vendors, delivering broken builds not as frequent as the vendor stated above :)

Return on Investment, in its true sense.

The ROI is not about the speed with which a certain task is done. The actual ROI is that the testing team time is now freed from checks to do exploratory testing, bettering the test coverage, faster feedback, better informed decisions, focusing on things that they were not focusing on AND preventing bad builds from entering the client's location.

I thank my colleagues (nicknamed) Rag and Tham for the support they offered me in the three weeks. Without a management providing freedom to a tester like me, this wouldn't have happened. So, thank you Mr. Head of Testing, Test Managers and Business Analysts for all the support and for approving this blog post.

The only thing pending is a party to celebrate. Mr. Head of Testing, are you reading this? :-)

27 comments:

Thank you for a great story! Really like how you cut the long story into pieces and extracted the real and good stuff from every piece. You got a whole lot of points that are really important as I see them.

Pairing with developers, business analysts and customers is just so great. I speak about my experience with the developer/stakeholder here and here.

Using TDD when its not very obvious that is what is needed. It takes alot of effort and a little turn-around thinking to use it properly.

Testability being a huge issue is also something that we are adressing at my client at the moment as well.

And then of course, giving the developers the incentives with real numbers about how much they will help by doing those small tasks to move testing forward with great leaps. I am going to take that with me and try to use it more in my team =).

Thank you for inspiration!Regards, Sigge

Btw, you have a small bug in your post. Do you find it? Well, this sentence is shown twice when referring to James and Jonathan.

Btw, you have a small bug in your post. Do you find it? Well, this sentence is shown twice when referring to James and Jonathan.

Thanks for informing me about it. I have made the correction.

Thank you for a great story! Really like how you cut the long story into pieces and extracted the real and good stuff from every piece. You got a whole lot of points that are really important as I see them.

I wrote this post for 8 hours and edited it for about 4 hours. It appears that the effort was worth it. Thank you.

Using TDD when its not very obvious that is what is needed. It takes alot of effort and a little turn-around thinking to use it properly.

Yeah, I tried the TDD idea and it was helpful in this context. We hear a lot of good things and hardly put anything to practice. I must say reading Edward De Bono's "Think - Before its too late" excited a few more neurons in my brain.

Pairing with developers, business analysts and customers is just so great. I speak about my experience with the developer/stakeholder here and here.

I am going to be reading your posts. It is exciting to read how others have done it because it would be helpful for my future consulting assignments.

Testability being a huge issue is also something that we are adressing at my client at the moment as well.

And then of course, giving the developers the incentives with real numbers about how much they will help by doing those small tasks to move testing forward with great leaps. I am going to take that with me and try to use it more in my team =).

Developers are rich source of information and help. After reading "More Secrets of Consulting - Jerry Weinberg", I realized that behavior of people around us is congruent to our behavior. The above case is very similar to that.

Inspirational and great story telling. You really get your points across well and your experiences will help me in my work.

The number of people involved and the networking you did shows how much better work we can do simply by talking to people and not rely on process and organization.

Establishing testabillity is of course paramount if you want to be able to test and for me this is one of the major tasks I have in my current project. We're in the early stages of development in my primary project right now and I daily remind the developers that I need logging and I need stored procedures, I need to get data that I can use from day one.

I love the "video scripting" for checking! I'll steal that immediatly, and let everyone know where I stole it. So easy, so simple and so very powerful.

I am very greatful to your client for allowing you to share your experience with the rest of us!

I am very greatful to your client for allowing you to share your experience with the rest of us!

Here was the conversation between me and my client:

"Would you permit me to blog about this experience if I hide all confidential information and get your approval before I publish?"

Response, "I think you should blog about this to help those people who feel blocked after just proposing a good idea"

I thank them.

The number of people involved and the networking you did shows how much better work we can do simply by talking to people and not rely on process and organization.

Processes were probably create to facilitate things but I don't think it ended up being a facilitator. I do see people being stuck in trying to follow the process - which is not their goal. I am not saying; all processes should be broken. I am asking people to consider challenging the process which restricts them from achieving a bigger goal.

With influential emailing, talking and body language, I could see people responding to what I require as I helped them understand what value I was trying to bring in.

Establishing testabillity is of course paramount if you want to be able to test and for me this is one of the major tasks I have in my current project. We're in the early stages of development in my primary project right now and I daily remind the developers that I need logging and I need stored procedures, I need to get data that I can use from day one.

I would prefer to make a presentation to them of how much less I would disturb them if they invested just a little bit of time to get me these things followed by their favorite drink or pizza. Not that we are bribing them, we are showing care towards a common goal we have.

I love the "video scripting" for checking! I'll steal that immediatly, and let everyone know where I stole it. So easy, so simple and so very powerful.

Here in India, many managers and tester cry over the need to provide their client an evidence. I have been telling them about videos for evidence and video test cases to show what we are exactly doing and for bug reporting.

If we are writing test cases for evidence, what can beat the evidence of video? That was the question I was trying to solve.

If a product has a button "Setup" in 10 different places and in 10 different contexts and some within the same page, which button should the newly joined tester click if he reads through a test case written by someone who is poor in English?

I am proposing my client the idea of videos to be sent for in country testing or on site testing by business analysts.

Maybe you would tell me your story soon.

I am not permitted to share everything but I am glad I could share most of the important ones.

BTW, thanks for making my ideas your own. That's a way to move forward.

A big thank you for sharing your testing experience Pradeep. Its indeed great learning material for me!Reaching out to people and articulating ideas, requirements does help a lot more than just formal requests or emails.I'm all for team collaboration and team work.I vaguely remember having done some video scripting, but it was to demonstrate product behavior in a particular test scenario and not on a regular basis. But it sure will be much more interesting and usable than test cases in an excel sheet!And testability needs to be woven into the fabric that is the product! makes a testers life that much more easier and for that matter, a developer's too! i think it all depends on the development standards followed.And thanks so much for sharing the email! Composing an email is an art i believe. and though we learn to better it with time... its great to see such an example!

I vaguely remember having done some video scripting, but it was to demonstrate product behavior in a particular test scenario and not on a regular basis. But it sure will be much more interesting and usable than test cases in an excel sheet!

Cool

And thanks so much for sharing the email! Composing an email is an art i believe. and though we learn to better it with time... its great to see such an example!

I have seen people not learning with time. I don't think they are putting in any efforts to improve on it.

A lot of reading, writing practice and getting it reviewed by people is required to develop that skill.

Perhaps I'm missing something but if you are following the sames steps that someone else has followed (and not deviating) then yes this could be called a check. Afterall you've put not thought into this and just followed someone elses steps.

Having said that if you then go beyond what the vidoes you followed, as you learn more about the software under test, and start to think for yourself and follow your own instinct and start going through steps you've thought of then you're testing.

You've got to start somewhere when learning about a new piece of software so from there your checks will develop into tests.

Love your description of how you convinced the other tester to use screen capture.

Perhaps I'm missing something but if you are following the sames steps that someone else has followed (and not deviating) then yes this could be called a check. Afterall you've put not thought into this and just followed someone elses steps.

To me what matters is the value I deliver to clients. In that context, what I have done here is to identify machine dependable checks and automated them to free the time for exploratory testing.

However, I'd like to know what you might have done in this context.

Having said that if you then go beyond what the vidoes you followed, as you learn more about the software under test, and start to think for yourself and follow your own instinct and start going through steps you've thought of then you're testing.

I did put my thoughts. As you might read above, I proposed the evaluation of quality criteria that they had not considered, tested for testability, usability, performance, reliability and more. Stakeholders were not interested at it given their time and situation.

You've got to start somewhere when learning about a new piece of software so from there your checks will develop into tests.

If you ask me today, I can explain to you the product without showing the product to you and can draw a picture of different sub systems and how they are connected. I can test for regression and analyze the impact of a fix in the product. If I can do all this in three weeks, do you consider that I have learnt?

Love your description of how you convinced the other tester to use screen capture.

Thank you

Addition to that: I want to help you by responding to any further queries you may have.

I have been reading your blog regularly since 2 months now. Still, I am not that much experienced in software testing field but I find your blog as the best resource which I can refer to for getting the best knowledge about testing.

At my place, we are only working on the very basic testing and I am not getting chance to do other things and other kinds of testing. So, I am referring to your blog as a learning class and I really appreciate the efforts that you are putting to enhance the knowledge of testers like me.

'I have seen people not learning with time. I don't think they are putting in any efforts to improve on it'This is a sad scenario. As for me... learning is continual and i have tried to internalize it at all levels... doesnt matter if it is personal, technical.

'A lot of reading, writing practice and getting it reviewed by people is required to develop that skill'Very true. Fortunately, i have worked with people who have been trying to improve upon their communication skills. Most would ask me to review the emails before sending it to the client. Not that i'm perfect but they considered me to be better than the rest on the team :)

Hi Pradeep,That was the most interesting article I read in recent days .Just emphasises one point that coordinated effort can take teams or organizations a very long way , unfortunately in most cases coordination is the thing which is most lacking rather than technical or domain knowledge.Thanks once again for such a wonderul post .

Video recordings sound interesting indeed. But it will be hard to convince an organization to use especially when they rely on hundreds of thousands of test cases. There is problem of capturing, storing while preserving data confidentiality.

Yeah, I liked it too. Something got screwed up with the images and blogger had some fixes for it. People reading my blog from their offices had emailed me that they couldn't see half of the content, so I had to change and experiment.

- You were hired to test a product having the business rules engine - Historical data was there- report viewer shows the report with historical data- Product developed was to comply the business rules and show the data same as historical data.-This product was new and there was no db with the new data post running of business rules.

Now my curious question(s)?

- what database you queried to extract the values to be validated against the legacy data? or- What data was considered final? - who created the this data? - Is it your checker tool did create the test data and then also did the validation of business rule as well??

you said: Using TDD, you were able to find the errors in earlier code as well

This is first time i have been to your blog and believe me not only you have a great testing brain, you are good narrator too...

the best part for me in this story was- "Had my colleagues at the client location been people who were active in learning, I wouldn't have sounded like an expert tester to them." thats the learning i'm taking from here...

I sneakily read your blog from office.May I know what you meant by Video test cases as in our company also we use Webex recorder in our office to record the steps we perform?is this the same or is video test scripting something different.

I was watching "The Shawshank Redemption" yesterday , was reminiscing you & the quote "Some birds aren't meant to be caged"

I think I love testing!Otherwise being happy (read drunk) after a office party and under a thick Management Accounting assignment due I wouldn't be writing a comment to your post after reading some interesting testing blog posts.

Pradeep, I know the importance of automating checks but still if I procrastinate doing that and spend my valuable time doing checks sometimes instead of witty tests... What would be the solution to my problem :(

Hi Pradeep,Just came across your tutorial on 'how to hire testers' at www.softwaretestinghelp.com and from there I landed here. It is amazing how much information you can provide on so many things related to software testing. Your opinions and insights are highly appreciated.Regards,Manas Tare.

As a tester who was first a customer and then an IT Professionl before entering the world of software testing I strive to challenge the traditional ways that folks approach testing and to get people to understand the what and why behind the curtain, the dependencies.

The concept of tests and checks I first ran across here - and give me terms for thinking that I have had in the past.

I find the most difficult requirement to capture is the intent of the testing. I do many different types of testing with many different intended outcomes - I don't always look for bugs - it depends on what is necessary.

I am not a person to leave comments, but I thank you and I thank the link here from James Bach.

Posts & Comments

Search this blog

Loading...

Copyrights

Tester Tested! by Pradeep Soundararajan is licensed under Creative Commons. You must owe credits to Pradeep Soundararajan when you copy paste anything from here by mentioning the name and proper linking to the post. You are not allowed to edit any of the post without permission. For permissions, write to pradeep.srajan@gmail.com