Tuesday, 14 December 2010

People who follow my blog may already know that I have an interest in psychology and how it can impact our thinking in relation to testing. Previously I have written articles on emotions , feelings and confirmation bias. I intend within this article to look at Cognitive Dissonance and how much of a help or a hindrance it can be for a tester.

There have been many studies on Cognitive Dissonance and if it really does exist however it would be best to describe what it is first. Wikipedia describes in it very simple terms:

The history of Cognitive Dissonance is accredited to Aesop’s tale of the Fox and the Grapes where the fox who wants to eat the grapes cannot reach them and as such to remedy the conflict that he really does want the grapes decides that the grapes are either not ripe or too sour, hence the title of this blog.

My thoughts on this conflict of beliefs or opinions and how we adjust to resolve the conflict is something that I feel could benefit testers especially when carrying out exploratory testing.

As testers I think it is important we recognise when we are experiencing Cognitive Dissonance and learn to ask questions before we make a decision to resolve the conflict. The problem with experiencing cognitive dissonance is that it is very easy to change our opinion and belief and suddenly we are drawn into the trap of confirmation bias.

As humans we do not like the feeling of conflict within our minds and we try to make a decision to resolve this conflict. Once we have made that decision we will try to justify the reasoning for making the choice we did.

For example if you had two areas to test and each area had the same level of important and you choose area A instead of B. You will now subconsciously favour choice A more than choice B. Now if someone comes up to you and says Area B is more important you have a conflict and your mind needs to give reasons as to why you choose A. You may make statements such as well at least Area A has been tested, it was important for me to test it. You will try to justify the reasoning for making what appears in your mind is the wrong choice.

When testing if we get two conflicting oracles and you need to make a decision on what is correct or the product owner needs to make a decision, cognitive dissonance could come into play. A decision could be made which is wrong. Only the problem now comes at a later date when you need to justify why you made that decision you will change your opinion or belief to make sure that the decision you made was the correct one.

It can become worse if you use a rating system to rate identical items. Imagine you are in a team and the team is given the task to rate features to be tested or developed. These ratings are then used to determine the order in which features for the product are developed. You then make a choice to test/develop your most highly rated feature, at a later date you get to rate the list again the item that you rated as next best has suddenly become of low importance even though it has the same value as the previous highly rated item. Making a choice after rating affects your value/belief/opinion of the item of same identical value and you start to score it lower to stop your uneasy feeling or cognitive dissonance. Project manager and team leads need to be aware of this since something of high importance could be downgraded by a team because of the issues of cognitive dissonance.

So how else can this affect testing?

I believe it can actually harm what we are testing. If the belief we have for the software we are testing in what it should be doing contradicts what it is doing and we adjust the conflict to justify what the software is doing there is a danger we could miss an important bug.

Whilst I agree with Michael that testers need a wide variety of skills it got me thinking about skills that all testers need or should have. One of these is being able to deal with the vast amount of data that everyone has to deal with. How do we deal with?

I have recently been re-reading the classic HG Wells story “The Time Machine (Link) and started to think about the two communities with the story, the Eloi and the Morlocks and how society in general is getting so much information that it is starting to make us dumb, this IMO is a dangerous thing for testers.

Are we as a society becoming like the Eloi?

They have access to amazing technology that helps with all their needs however they come across in the book as dumb and lack curiosity. They see no need to be thinkers or philosophers. It appears technology is making things easier and easier for us.

When I was young (in the very old days) to find anything out I used to have to read a book. Since books were an expensive item I used to have a list of books I wanted for my birthday or Christmas and I used to visit the local library every week to sit and read and learn new fantastic things. I would get lost in a world of fantasy and knowledge; even then I had a thirst for learning which fortunately has never left me. It is such a shame that local libraries all over the world are shutting due to technology. (Do a Google search for News and Libraries and Closing)

Now information is available at the click of a button. We can find information on how an airplane works or the theory of relativity in an instance. Technology has made all this information easier to get, however IMO it has made us think less.

Do we still question all this information?

As testers we know we should be questioning everything, we learn to sort the chaff from the wheat as the title of this blog post implies. (Link). With such a wealth of information that is so easy to gather the skill is to be able to collate this information and remove the distractions. How easy as testers do we find this? I find it a natural thing I appear to do without thinking until I started to write this blog article. My concern is that with so much information do I end up throwing away something that is later proves to be vital or important.

Does anyone out there in the testing community have a method they use to help with this?

How do we not forget everything?

This then leads on to the topic of self learning – how do you select what to read and what not to read? How do you ensure you do not miss a really important article that has been blogged?

One approach I use for self learning is to use twitter and the software testing club (Link) within these communities’ people we talk about blogs they have read or recommend to read, the power of the crowd. This helps to reduce the amount of information I have to process. Another approach is to actually talk to people; humans are wired to be better at absorbing information via speech than from the written word, it is more likely to be remembered.

My other concern with all this information is our ability to remain focused, another important testing skill. With so much information to digest it is so easy to get into the habit of just scanning the information and not reading the whole article (I wonder how many people will get to this part of the article?). It is so easy to just start an article and get distracted by some other piece of information and not return to the original article. I sometimes think I should not add any hyperlinks to my articles and just add them to the end but I want to credit the people who inspire me or provide me with information as I write the article, it is one of those things which is important to me. A perfect example of this was the recently article Michael Bolton wrote about estimation (Link) which was a five part article, how many people read the whole of the article? It is so easy to skip or scan and miss an important point within an article and the same can be applied to testing. If we scan and miss something it could be that the thing we missed will cost us a lot of money.

My other concern is that we are becoming a society of 24x7 learners, we never switch off.

Are you one of these people?

My concern on this came from a conversation in which I stated I do hobby as a job, this scared me. I have a passion for testing and learning but am I not in danger of burning myself out or forgetting valuable knowledge unless I switch off?

How many others reading this blog switch off and pursue other interests outside of technology?

If you take nothing from this article please do switch off. I have hobbies that have nothing to do with computers. I enjoy being creative and take photographs, spending time outside at stupid o’clock catching sunsets and sunrises and landscapes. I enjoy growing things and spend time in my garden. I am fortunate to have a very large garden in which to grow and nurture things. I also have a family and I am a grand parent and spending time with my granddaughter is such a wonderful thing to do. We call her our little time waster – since time can go so quickly when you are engaged in playing. After all these hobbies it is surprising that I have time to do my job or to keep learning but I come back to my work more energized and ready to learn more.

Since starting to write this article I have found a couple of other blogs that mention the problems of attention span and remaining focused they can be found here:

The other highlight was amount of ‘real life’ examples of exploratory testing and session based testing management. One of the best things I took away was from Carsten Feilberg’s talk on Session-Based Testing in Practice (http://carstenfeilberg.blogspot.com/) in which he reframed the wording of SBTM to Managing Testing Based upon Sessions. It was a why did we not think of that before!!!!

One of the keynotes was by Stewart Reid on “When Passion Obscures The Facts: The Case for Evidence-Based Testing” in which he looked at what testing could learn from Evidence Based Medicine (http://en.wikipedia.org/wiki/Evidence-based_medicine) . During the presentation I thought I could see many flaws in the argument he was trying to put together but could not quite work out what it was. One thing I have found out since and one point that Stewart did appear to miss was the work of the GRADE Working Group which is a newer system (and appearing to gain ground). The principles here are based upon Extrapolations (http://en.wikipedia.org/wiki/Extrapolation).

To quote from Wikipedia:

Extrapolations" are where data is used in a situation which has potentially clinically important differences than the original study situation. Thus, the quality of evidence to support a clinical decision is a combination of the quality of research data and the clinical 'directness' of the data.

Interestingly the data gather for extrapolation are more based upon human experience rather than just a set of numbers. Is it just me or is this like running a set of known tests then exploring afterwards? See my previous post on Hybrid testing (http://steveo1967.blogspot.com/2010/09/hybrid-testing.html)

So why have I called the title of this blog post “The Human Element”?I was having a conversation with my wife(Tracy) after the conference since she is a retired theatre nurse and understand the medical arena very well and she came up with a wonderful phrase. It is all well and good having all these numbers and statistics but you cannot ignore the human element. She gave an example of this in which a nurse working in Intensive Care has a lot of machinery (with installed software) at her disposal however none of this equipment can tell her if the patient is feeling happy or sad or is uncomfortable.

Tracy said the problem is no machines have a soul they do not care how the patient is feeling, the machine could be saying everything is ok but the nurse and their compassion knows and understands how the patient is. I asked my wife to have a talk with Stewart and some other testers including Lynn Mckee (http://www.qualityperspectives.ca/)

This provided a wonderful insight to me in that we as testers forget that there are lots of people who we should be using as oracles for when we test a system we should not be forgetting about the human element.

During her conversation Stewart started to mention the use of statistics as evidence and for making healthcare decisions (Cochrane Library - http://www.thecochranelibrary.com/view/0/index.html?s_cid=citation) and Tracy said that to get to the point of making a decision still requires the GP to ask questions and to explore all possibilities. At the end of the day it is just statistics said Tracy and it does not help in a situation in which a perfectly healthy 20 year old is prescribed a drug for a problem and then dies due to an undetected heart problem. No amount of journals, evidence can account for this, since it is on a personal level between the patient and the medical expert.

The final conversation I remember Tracy having was with Lynn and a few other and it is very useful for testers when they come up against the problem of ‘It should do this.’Tracy talked about Dr Spock and the book about the development of children (http://en.wikipedia.org/wiki/Benjamin_Spock) and that at certain age’s children should be doing this and that. This book causes major worries in parents when their child does not meet the timescales within the book for talking, sitting up, walking etc. Tracy then made a point which caused a great amount of laughter. “People seem to forget that babies have not read the book – they will develop at their own pace”

I found this a wonderful piece of insight, we seem to forget that everyone is different and if we apply this to software and the development of software we start to realise that every piece of software is different and that we need to explore the software and play with it to get the full potential out of the software.

To conclude this post I would like to say a big thank you to my wife Tracy for her encouragement and support in what I do and for giving us testers a lesson in remembering about the human element in what we do.

Tuesday, 2 November 2010

I have noticed that I have been a little remiss with my blog recently, this has been due to a combination of different things such as workload, home life and not having a great amount to say, which is fairly unusual for me. I don’t want to blog for the sake of blogging I want to blog when I feel I have something to say about the testing world.

I will soon be on my travels again to talk about exploratory testing and testing skills this time in Israel as part of an internal company workshop. I find it interesting that again I will adjust my material to match my audience on a cultural level, see my previous blog about training in India (http://steveo1967.blogspot.com/2010/06/training-in-india.html).

I wonder how many of us do this and how many of us just keep the same material and just recycle it regards of the audience?

This brings me to the point of this blog, if we treat software as different cultures and we try to explore and communicate with these cultures in the exact same way each time without making adjustments for the cultural differences.

Are we going to get to know anything about this culture?What will we learn?Will this culture give us any useful information back?

If we compare this the approach I use when presenting you can see that I learn about the culture. I am exploring by communicating with it and finding out about all the subtle differences there are. I try to avoid the traps and fopars that can cause offence by being ignorant of the culture. I consult oracles that have knowledge of the culture; I use heuristics when presenting to test the reaction of the audience.

Does it respond well to what I am saying?Is it losing interest?

I then adapt my presentation on the fly to try to re-engage with the audience.I am using the exploratory approach when presenting and I do this naturally. I very much doubt that most people who read this blog do not change their material, communication methods and approach when working with different cultures.

So why do we as a testing profession still insist that we can test software with scripts that do not change or adapt to the slight differences in culture? Yes there is an argument that says something’s do not change regardless of the culture and that is true. However if you go to a different culture and you are ignorant about their values and beliefs and you are unwilling to learn then you will leave that culture none the wise or richer in experience and understanding.

Do not be ignorant about exploring software, yes you can use the same techniques and methods you have gathered over the years to explore the software but do not fall in to the trap of following things by rote.

Hopefully there will be a few events coming up soon in which I can get some more topics to blog about.

If anyone is interested I will be at Eurostar (http://www.eurostarconferences.com/) this year in Copenhagen Denmark and it would be nice to meet up with like minded people and hopefully have some great discussions over beer of course.

Monday, 27 September 2010

There has been a lot of talk within the testing community about the scripted v non-scripted approach to testing. I have read and heard from people aligned to each school of thought trying to debunk the other schools approach. This can be very confusing to those that work in the profession of software testing. There are arguments on either side with people on either side presenting their point of view. I thought I would blog about my experiences of using both approaches in my day to day testing job.

When I first started in testing I worked for many companies which had adopted the Prince 2 methodology of software development and loosely followed a V-model process. This meant that requirements and specifications were gathered before any development work started. Using these documents as a tester I would do some gap analysis from a testing perspective to see where requirements contradicted each other and design specifications did not meet requirements. These were very heavyweight documents and it was a laborious task that engaged me as a tester to a certain point. Using these documents I would start to create scripted tests and build up a repository of test suites. Once the software started to be developed and I could gain access to certain features my engagement as a tester increased. I would run through my scripted tests and find that a large amount of them would need altering since I had made the wrong assumptions or the requirements and specification did not meet what was delivered. As I ‘explored’ the software I found more and more test ideas which would become test cases. The amount of discussions I had with senior management on why the number of test cases was increasing is another story altogether. I would spend a large amount of time adding detailed steps to the test scripts and then when we had another drop of the software run them again as a regression pack. I tried to automate the tests which for some easy parts worked and for others did not. What I did not know at the time was I was carrying out exploratory testing without knowing I was doing so. Once I had the software it was the most engaging time as a tester, it was what made me feel like I had done a good job by the end of the day.

So let us jump forward to today: - we have TDD, agile and a multitude of different approaches to software development. It is all about being flexible and developing software the customer needs quickly and efficiently and being able to adapt quickly when customer needs change. As testers we get to see and explore the software a lot sooner.

A lot has changed from a tester perspective we are now engaged more in the whole process, we are expected to have some knowledge of coding, IMO not always necessary but a good tool to have. We get to see the software a lot sooner and able to exercise and explore the software and to engage our testing minds to what the software should, could or may do. However have things changed that radically?

What has made me think about writing this blog has been the debates that have been going on about scripted vs. non-scripted. I am currently working on a new project in which there are many dependencies on internal components and external 3rd parties all of which are working to different timescales. Some of the components can be simulated which others cannot due to time constraints and other technical problems. We have some pretty good requirement documents and some design specifications. What we do not have at the moment is fully working end to end software. So I am back creating scripted test cases to meet the requirements, finding discrepancies in the documents and asking questions. The difference is that now I do not fully step out my scripts I create pointers on how to test the feature, I note test ideas that could be interesting to look at when the software arrives, I make a note of any dependencies that the software would require before testing that feature. So I create a story about testing the feature rather than create a step by step set of instructions. It is more a testing checklist rather than a test script. So with this I am combining both scripted and the non-scripted approach. I am sure a lot of readers will read this and think that they are doing the same.

The people who talk about exploratory testing have never said to my recollection that there is no need for scripted tests. Some of the requirements I have are fixed they will not change; the results should be as stated; so those requirements I can script or automate if possible. It does not mean you are doing anything wrong nor does it mean that you are not following an exploratory approach. Exploratory testing is just that an approach, it is not a method, it is not a do this and nothing else. It is a tool to use to enable you to test and hopefully engage you in testing rather than just being a checking robot. If you still create detailed step by step scripts then there is nothing wrong in doing that, I still do when required.

Exploratory testing can be used without the software, you can look at available documents and explore them for test ideas and new creative ways to test what the documents are stating, you can question, you can analysis you can make suggestions and improvements, you can use your brain.

Wednesday, 8 September 2010

I have noticed that it has been awhile since I did a blog due to family commitments and vacation time. I had an idea to blog about the importance of gathering evidence when testing especially when using an exploratory testing approach. I decided to take the example of why it is importance from an internal workshop that I run on exploratory testing.

So are you sitting comfortably?

Here is the story of Timmy the Tester and Elisa the Explorer

Timmy Tester

Timmy Tester has been given a new build to test and decides he is going to test it using the exploratory approach. Timmy writes that his mission statement is to test function x has been implemented.

He installs the release and starts to enter different values and pressing buttons around the new feature at random. He does this for about half an hour and then all of a sudden the application crashes.

Timmy takes a screen shot of the crash and a system dump of the crash and goes to talk to the developer. The first question the developer asks is

“Have you tried to reproduce the problem?”

At this point Timmy says no and goes back to try to reproduce the problem.

Two days later Timmy has been unable to reproduce the problem and now thinks it could have been one of those strange things.

3 months later the application is now live on the customer site. Within 30 minutes there are a large number of calls to support stating the application is crashing. The problem gets passed back to Timmy who notices that the problem appears to be the same as the one they saw when carrying out ‘exploratory’ testing….

Elisa the Explorer

Elisa has been given a new build to test and decides she is going to test it using the exploratory approach. Elisa creates a mission statement stating they are going to be testing the new function.

Elisa installs the new application and starts to enter different values around the new feature. As she is doing this Elisa has another computer in which she makes notes and takes screenshots at relevant points to aid clarity of each step that she has carried out. At certain points Elisa finds behaviour of the system which does not seem correct so she starts another mission statement to look into that behaviour. Elisa then starts to examine the strange behaviour in more detail making notes of the steps she is carrying out at the same time. All of a sudden when pressing a button the application crashes.

Elisa makes some notes, takes a screen shot and a system dump of the crash.

Elisa then resets the application back to a clean system and repeats the last set of steps which she had made a note of. The crash happens again.

Elisa then goes to see the developer and states she has managed to produce the problem more than once and here are the steps.

Elisa sits with the developer while they go through the steps together and the developer sees the crash.

Two days later Elisa has been given a fix for the crash. She now has an automated test for the crash and runs it straight away. The test passes and Elisa continues with the rest of her testing.

It may seem like common sense but I have seen more cases of Timmy than Elisa from people who have said they are using exploratory testing. It is extremely important to record everything and remember exploratory testing does not remove any of the principles of testing:

“All tests are repeatable”“All problems are reproducible.”

There are many ways we can gather evidence of our testing sessions and there are a large amount of tools available to the exploratory tester. In much the same way that when the first humans decided to explore the North Pole they took tools with them that could help and support their efforts exploratory testers can do the same when exploring the system under test.

Maybe I should look at some of these tools and write a blog about them – or even better people who read this blog might be able to suggest some good tools that they have had experience of using.

Monday, 2 August 2010

I had a few moments to myself the other day and decided to have a bit of fun and do some research about what agencies required when you apply for testing roles. I was surprised (or not) by the number of roles that stated “will be ISEB certificated”

How can they get it so wrong? The ISEB is now defunct it should be ISTQB (sic) so all those who hold an ISTQB qualification need not apply………

Anyway back to the point I am trying to make.

I did some quick research and found that within the last seven days of the 122 testing roles listed I would be excluded from 28 of them since they stated …will be ISEB certified. If you add in the ones that add ‘would prefer candidates with ISEB certification. You get to over half of the roles advertised that I could not apply for.

@Mheusser made the following tweet @steveo1967 - certification might just get you the job you don't want to have.

That maybe be true but I got thinking and maybe it is not the company using the agency that are mandating this but the agency adding its own filter. The company could be missing out on some great testers because of the agency filtering system.

The following are some imaginary conversations with an agency (Sadly I am sure some people may have already experienced this…)

Me: Hi I am interested in being put forward for the role of chief tester as per the advert you posted this morning on your website.

Agency: Sure, have you already sent us your CV, Resume?

Me: Yes

Agency: OK, what is your name, let us see if we can find your CV

Me: My name is Timmy Tester

Agency: Wow cool name sure matches your profession

Me: Yes (Sigh) I get that all the time..

Agency: OK Got your details – just looking at them now, wow 20 years in software testing, that is impressive, I see you have worked at some very well known companies. Oh wait a minute…. You do not list that you hold the ISEB qualification.

Me: No I do not.

Agency: I am sorry Timmy I cannot put forward your application – we only accept people who are ISEB qualified.

Me: Why? I have more than 20 years in software testing.

Agency: Yes I can see that, however the ISEB qualification means that you know how to test and is mandatory for any roles you apply for with us. Sorry but I cannot put you forward for this. I suggest you go and sit the exam and get back to us. Goodbye.

Me: Hi I am interested in being put forward for the role of chief tester as per the advert you posted this morning on your website.

Agency: Sure, have you already sent us your CV, Resume?

Me: Yes

Agency: OK, what is your name, let us see if we can find your CV

Me: My name is Ivor Certificate

Agency: OK Got your details – just looking at them now, you have been in testing for 8 years now, I see you also hold the ISEB qualification. That means you must know a lot about testing.

Me: No I do not.

Agency: Sorry, you said you do not?

Me: Yes I did say, no I do not know anything about testing

Agency: Then how come you have a ISEB Certification?

Me: Because I noticed that if I had this I could apply for any testing job.

Agency: So you must know something about testing?

Me: Well it is an interesting story. I paid to do the multi choice exam, sat down and completed it in 10 minutes by just randomly answering the questions. No one checks if I have any competence at testing, by luck I managed to get the pass mark required to get the certificate. Hence I am now classed as someone who must know about testing and how to test.

Agency: But it says on your CV that you have been testing for the last 8 years.

Me: Yes I have but I just go in and let others, who are not certified do all the work while I just copy what they have done and claim credit for it. So are you going to put me forward for the role?

Agency: Yes I will you meet the selection criteria, so I cannot see any problems in putting you forward.

This may seem like a silly situation but I am sure it could happen in reality. I am not against qualification and people trying to improve themselves but when those qualifications are then used as a filter to exclude people from applying from jobs, it makes me see red.

There are some good examples of proving your ability as a good tester. You can talk to previous companies that you have worked for. They could interview you and talk to you about testing and your thoughts, problem solving ability. However this would take too much time.

I have not as yet found any roles that have stipulated that they require that you attended a Rapid Software Testing Course or The Black Box Software Testing course through the Association for Software Testing. Why is this so? Is it that these types of courses do not have the huge budget to promote themselves? Or that they try to be non profit making?

What is the solution to all of this?

I think within the testing community we need to start educating agencies and companies about how to sort out good from bad testers, how we go about this I am not sure.

Should we have a dedicated website that we can direct agencies to, to explain about certification? I feel this could be a good start and would need someone with far better web skills than myself to get running and also it would need to be unbiased as possible.

Then mail shot the CEOs at each agency when we hit this problem directing them to the website.

Do we try to do presentations at employment agency conferences?

I feel there is a need to educate agencies and companies that are looking to employ testers and give advice on how to spot the good from bad candidates but they need to get rid of the certification filter.

Another thought I had would it be a good idea to have a vetting service for testers and agencies? There could be a one stop service for agencies to verify testers, their abilities and obtain a list of people who would vouch for them.

I tried to think at the weekend if this could work or not. I have a lot of concerns about it being misused and ‘gamed’ by people who have a moral compass that is slightly off balance. How would it be funded? Would it become a monster of its own making? How would testers be vetted and vouched for? Would testers be vetted and vouched based upon their online presence? For example I am sure I could ask a few people online to vouch that I am a good tester however none of these people have worked with me and seen me carry out testing. I could be just saying the right thing at the right time to impress people – how would anyone know unless they have worked with me?

This really brings us back to the beginning of the article, agencies and companies need someway to vet testers and get some guarantee that they know about testing (regardless of which school of thought they follow). So using the certification method is an ideal way to sort out candidates quickly no matter how flawed the certification may be. Do I just bite the bullet and sit the exam if I do not want to be excluded for any testing job? Does anyone have any other methods that agencies or companies can use when they are looking for skilled testers?

Each of these blogs makes very valid points about how non-useful measuring test cases are for indicating the progress or coverage of testing effort.

The aim of this blog is to try and expand upon these posts and see if there are ways in which we could measure testing effort and progress within resorting to using numbers.

To start with we shall take a look at a made up situation.

You are testing on a project which is has a two week testing cycle, your manager has requested that you need to report each day the following:

How many test cases you have

How many have been run

How many have passed

How many have failed.

(Does this seem familiar to anyone?)

So before you start testing you report to your manager that you have 100 test cases to run over the two week cycle

At the end of day one you report the following

Test cases ran: 60

Test cases pass: 59

Test cases fail: 1

Defects raised: 1

Test cases still to run: 40

So management think cool we are ahead with testing, 60% done in one day.

At the end of day 2 you report:

Test cases ran: 62

Test cases pass: 61

Test cases fail: 1

Defects raised: 1

Test cases still to run: 138

Management now thinks how come you only ran two test cases today, why are you running slowly? WHAT!!!! Where did those other 100 test cases come from? Did you not do your job correctly to begin with?

However the two you ran today had lots of dependencies and very complex scripts.

Plus your testers noticed that there appeared to be new features that had not been documented or reported, you have now had to add another 100 test cases. Also your testers actually think when they are testing and thought of new edge cases and ways to test the product whilst they were testing.

Management starts to panic – you reported on day one that 60% of testing had been completed. Now you are saying only 30% of the testing has been completed, Stakeholders are not going to happy when we report that we have only covered 30% when the day before I reported to them that 60% had been completed.

This continues, your testing team are really good testers and find more and test ideas which are turned into test cases. So at the end of day seven you report the following:

Test cases ran: 1200

Test cases pass: 1109

Test cases fail: 91

Defects raised: 99

Test cases still to run: 10000

So at the end of the first week you have only completed 8% of all the test cases. You get fired for incompetence and the project never gets released.

Many people reading this may have experienced something similar to the above, what worries me that there are still people stating the best way to measuring testing is by the use of test cases!

The question now is that if measuring by the use of test cases is not a good way to measure then what can we do?

The following suggestions are my own and what I apply within my test approach, it does not mean it will work for everyone nor am I saying it is the best approach to take. However the purpose of my blog is to offer suggestions about testing that could be useful to some people.

I work in the following testing environment:

Agile based – 2 week iterations

Customer changing requirements frequently

Code delivered daily

Functions and features added without supporting documentation

Use a mixture of scripted and exploratory testing

If I tried to report the testing effort using the traditional test case scenario it would be of little (or zero) value, since the test case number would be constantly changing.

What we do is split functions, features etc into test charters, as per the exploratory testing approach, these ‘Test Charters’ are known as the test focus areas of the software. If a new function or feature is discovered a new charter is created.

We then use the Session Based Test Management approach (James and Jon Bach - http://www.satisfice.com/sbtm/) and implement sessions based upon mission statements and test ideas. During the testing session the testers are encouraged to come up with new test ideas or new areas to test, these are captured either during the session or during debrief.

The reporting of progress is done at the test charter (test focus area) level. The test manager reports in the following way.

Test focus area 1: -Testing has started – there are a few issues in this area:

Description of Issue x, issue y, issue z.Which need to be resolved before there is conference that is area is fit for its purpose.

Test focus area 2 – has been tested and is fit for it purpose

Test focus area 3 – test has started and some serious failures have been found defect 1, defect 2, defect 3

And so on.

Some people may ask but how will this tell us if we meet the deadline for testing? I am sure it will NOT tell you if you will finish ALL of your testing before the deadline since testing is an infinite thing, we as testers will carry on testing until we meet a stop heuristic (See Michael Bolton article on stopping heuristics: http://www.developsense.com/blog/2009/09/when-do-we-stop-test/).

The problem with testing is that it is not a yes or no when it comes to the question of have you completed your testing yet. Every time a skilled tester looks at the software they can come up with more and more test areas and test ideas that they could carry out. These may or may not add benefit to the suitability of the software and if it is fit for its purpose. What is required is a test manager that talks to and listens to their test team and see which test areas are the most important and MANAGE test sessions based upon what is critical – basically do some good old prioritizing. The test manger needs to ask the difficult questions of the stakeholder and project managers.

What features can you do without?

What are the critical areas that are required?

Function abc has many serious problems – it can cause problems x,y,z for your users. Do you need function abc?

We have tested all the key functions and found the following problems x,y,z. You want to release tomorrow, are you OK with these known issues?

In return the stakeholders and project managers must trust the test team and accept that when they report that an area has been ‘sufficiently’ tested they believe them.

To summarize – instead of reporting on a small area of testing such as test cases, move a couple of level ups and report on the progress for test areas/functions./features based upon the importance of the feature. This may not tell you if you will compete the testing before the deadline but it will show you how well the testing is progressing in each functional area at a level that stakeholders can relate to and understand. The trust your stakeholders will have in you should improve since you are giving them a story about the progress of the testing effort without trying to hide things using numbers.

Tuesday, 27 July 2010

In my previous blog I touched upon a term called Confirmation Bias and how as testers we should be aware of this. I stated that I would put a blog together on the subject so here it is.

I should start by defining what confirmation bias is.

Confirmation bias refers to a type of selective thinking whereby one tends to notice and to look for what confirms one's beliefs, and to ignore, not look for, or undervalue the relevance of what contradicts one's beliefs:- http://www.skepdic.com/confirmbias.html

A good example of this is if you are thinking of buying a new car and all of a sudden you seem to notice lots and lots of the model of the car you was thinking of purchasing. You mind is conditioning itself to notice this make and model of car and making you notice them more, even if there are no more than there was before – you appear to be seeing them everywhere.

Another example is if you start talking to a friend about a certain film and actor and then suddenly notice lots of coincidences, the actor is on a advert, the film is being shown again on TV, a support actor is in another film you just started to watch. The following gives a good example of this. http://youarenotsosmart.com/2010/06/23/confirmation-bias/

If there was no such thing as confirmation bias there would be no conspiracy theories. Conspiracy theories are based upon information which proves the theory correct; those who believe in the theory ignore the evidence that debunks that theory.

So why is there any concern for testers?

Let us start with an example.

You are working closely with the development team and you start to ask them questions about the release you are about to test. You ask their viewpoint on which areas they feel are the most risky and which they feel are the most – so you can adjust your priorities as required, a pretty standard exchange between developers and testers. You now start testing beginning with the area of high risk and work your way to the low risk areas.

You find a few serious bugs in the high risk areas (as expected) and you find no problems in the low risk areas.

After release a major bug is reported in the low risk area you tested. How did you miss the bug? Did you see the bug but your thinking was that everything was working alright? Did confirmation bias play a part? Did your subconscious hide the bug from you? Now this gets very scary, most people who work in software testing know that some bugs try to hide from you, we expect them to hide in the software. What happens if they decide to hide in your brain?

So how can we try and prevent confirmation bias?

The quick and easy way to try and prevent confirmation bias is to ensure that more than one tester tests the same feature, they may bring in their own confirmation bias but hopefully it will be different from the previous testers bias. There is more chance that it will be different if the testers have not discussed the area under test beforehand.

Another way to try and prevent confirmation bias is to do ‘paired testing’ either with a software engineer, another tester or a user. That way you can question each other with regards to what is true and what is false. There is a chance that you could cross contaminate each other with your own confirmation bias, but the risk should be less than if your are working on your own.

It is not easy to remove confirmation bias since it is infectious. The way of working on a software development project requires testers to communicate more and more with other areas of the business and at each stage and with each conversation confirmation bias could be introduced.

So should we lock ourselves away in a dark room with no communication with anyone else on the team? I think I would get out of testing as a career if that happened, the Social Tester (@Rob_Lambert) would now be the anti-social tester, time to get him a ASBO (For our non-UK readers - http://en.wikipedia.org/wiki/Anti-Social_Behaviour_Order)

My view is that there is no realistic way to prevent confirmation bias due to the way software development projects work and that there is a need for everyone to be able to communicate with each other. However if testers are aware that there is such a thing as confirmation bias then they can try and take steps to ensure it does not creep into their testers. That is the whole concept and point of this blog – to help to raise awareness of confirmation bias and how it can effect your testing.

Monday, 19 July 2010

The first part of this blog looked at how our emotions could affect how we test. This second part will look at how we could capture our feelings when testing and could this provide us with any useful information about the product we are testing. Could it prove to be a useful oracle when testing?

On twitter @testsidestory said the following:

That is done regularly in usability labs: capture emotions and facial expressions of the users as they use the s/w

This was in response to a question that I posted on twitter:

…. - what I am thinking is that we need to capture our mood when #testing it could indicate a problem in the s/w…

The concern with this is that it would be very expensive to implement for the majority of people. I thought how we could implement a system that could capture emotional state and be effective and inexpensive.

One idea I had was to use a concept from the book Blink by Malcolm Gladwell, in which Malcolm talks about how important our initial emotion/reaction is when we first encounter something. There is a discussion about how often our ‘gut reaction’ proves to be correct and he uses an example of a statue that a gallery had bought after a lot of scientific experts, who had tested the statue, had said the statue was genuine. A couple of art experts who got to see the statue before it was unveiled in private viewings had a ‘feeling; that there was something wrong about the statue, their initial gut reaction was telling them it was a fake. Several months latter it was discovered to be a fake.

The above is a concise retelling of the story within the book, however why did the scientific experts get it so wrong? Could it be that conformation bias played a part? The scientific experts wanted so much to believe that it was real and not fake they caused bias in the results or avoided obvious facts that pointed to it being a fake. I think confirmation bias is a great subject and one I will look at from a testing perspective sometime in the future.

So can we use this ‘gut reaction’ concept in testing?

Would it be of any value?

I should state that I have not tried any the following ideas and that if anyone would love to volunteer within their organizations to ‘trial’ the ideas out I would be most interested. Due to circumstances I currently do not have the ability to try this out on a large scale.

The first problem we face is how we capture out initial reaction to what we are testing. The requirements for this are that it is:

Easy to capture

Simple

Quick

My thought is to use different smiley’s which are simple and quick to create and capture thus covering all the requirements.

My idea would be to use three different smiley’s:

Happy

Neutral

Unhappy

Why use smiley’s?

The idea as to why use smiley’s is that anyone can draw them no matter how artistic and from the perspective of measurements it is very easy to recognize and see pasterns when using such well known symbols. The other longer term thought was that it is easy to extend to add sad, angry, and extremely happy if you wish to improve the range of emotions and feelings.

Capturing the initial feeling/emotion.

If you are working in an environment in which you are carrying out exploratory testing and following mission statements (Session based testing) then this is very simple to implement. The idea is that when the tester starts their mission (session) they should within the first couple of minutes (5 at a max) record their emotion/feeling of the software by the use of the smiley’s.

If this was done for every session being run and captured in such a way that it would be easy to see at a glance which areas (test charters) testers are unhappy with it could provide some useful information.

So you now have a whole set of data with regards to the testers initial feeling about the software there are testing, what does this information tell you?

For example a certain test focus area shows that all the testers are unhappy in that area would this indicate a problem? I feel it could indicate something wrong in that area but you would need to talk to the testers and gather more information (obtain context) I think the great thing about capturing initial feelings towards the software could help the development teams to focus on areas where there could be implied problems based upon initial feeling.

This approach could be taken a step further and get the testers to add another smiley when they have finished the session to see how they feel about the software after they have finished their session. You now have two sets of data and can compare any discrepancies with the two.

What would you think if the majority of testers were happy about a certain test focus area but at the end of the session they were unhappy?

Does this indicate a problem?

Or what if it was the opposite mostly unhappy and at end of session they were happy?

Also if they were unhappy at the beginning and at the end, their gut reaction proves to be correct, does this give an indicator that there are some major issues within that area?

Could this indicate frustration with the system, lack of knowledge maybe?

In my opinion this approach could provide to be a very useful oracle to the quality of the software.

Friday, 16 July 2010

This blog is going to be in two parts, the first will focus on the question of do emotions affect the quality of testing. The second will look at ways in which we can gather information about how we feel about the product we are testing to see if there is any value in capturing this information.

I have an amateur interest in psychology and how some of the ideas and thoughts from this area of science can be used in software testing. I was reading ‘The Psychology of Problem Solving’ by Janet E. Davidson & Robert J. Sternberg and it had a section on how emotions affect the way we think and focus.

So I decided to tweet a question based on some of the information I had read:

Emotions and #Testing:-Do we find more bugs when we are in a bad mood? Psychology research shows we are more negative when in bad mood.

It would be interesting to have feedback from #testing community on this - Does this mean a good tester has to be a grumpy so and so... :o)

This turned in to a lively debate on which mood is better for testing.

After reading various articles there appeared to be some common ground on how we think and see things based upon our emotions and mood.

Looking at the article suggested by @ruudox this suggested that when in a good mood we can see the whole picture and when in an unhappy mood we narrow our focus.

This appears to be backed up by research from Foless & Schwarz

Individuals in a sad mood are likely to use a systematic, data driven bottom-up strategy of information processing, with considerable attention to detail In contrast, individuals in a happy mood are likely to rely on pre-existing general knowledge structures, using a top-down heuristic strategy of information processing, with less attention to detail (foless & Schwarz, 1999;).

This now leads to some complex dilemmas, and the whole point of this blog.

Which mood is best for someone whose is a professional tester?

Which mood is more than likely to find more bugs when testing?

What other influences can affect our ability to test well?

My thoughts indicate from the information and research I have read that to be really good at testing and finding defects we need to be in a sad or unhappy mood.

Research concludes that when in a sad or unhappy mood we are more than likely to focus in on the task and step though in a data driven way. When happy we are more than likely to see the whole of the picture and look at the task from a top down approach.

Now in my opinion both of these traits are needed to be excellent testers. So do testers need to have split personalities that they can switch on or off?

The point made by @nitinpurswani about being in a bad mood stops him thinking and that to be a sapient tester he needs to think. This got me thinking and I asked him a question back.

@nitinpurswani I like that idea. However if you're in a bad mood with what u are #testing would it make you want to break it more?

My thought behind this is that if something is annoying me or irritating me I feel I am more than likely work harder to find out why it is annoying me. I become deeply focused on the problem in front of me. Does this mean I am in a bad mood? Not necessarily so – it could be I am annoyed at what I am testing but not in a bad mood in general.

When in a happy mood when testing it is easy to just let things go, we unconsciously think well that is not too much of problem we can forget about it. This is a dangerous attitude to have as a tester because this simple little problem can come back to be huge problems. Someone in an unhappy mood is more than likely to investigate why this thing is annoying and find the underlining cause.

@Rob_Lambert made a very valid point that there are environmental issues that could come into play. How many testers when testing listen to music? Rob suggested that the type or style of music you are listening to can influence the mood you are in and as a side effect the way you are thinking. I had not thought about this very much but going deeper than this – if you are working in a open office and everyone around you is having a laugh and joking would this make your testing better or worse? What if a tester and a developer are having a heated debate about something that has just been tested? Will this influence your testing?

Does any of this article back up my earlier tweet that testers need to be grumpy so and sos?

However I think this view is too simplistic. I am often asked about testers and how they are different from developers. (There is still a big drive within testing that developer and tester can be the same person and be able to switch between the different roles). I have a feeling that some of the best testers can switch between different psychological emotional states when testing. They have the best of both worlds. Able to remain focused when something is bugging them and then when they have solved what is bugging them able to switch to a whole picture view of the system they are testing.

When I started to write this article I thought it would be very simple to come to a conclusion about how emotions can affect our ability to test and what is the best mood to be in to get the best out of testing. It has proven more difficult than I thought and I still have not come to any firm conclusion about which is the best.

The one interesting point that should be made is that as professional testers we need to be aware of our emotions and how they can affect the quality of the testing we are doing. Part 2 of this blog will be looking at how we can capture our emotion and feelings about the product we are testing and see if this could provide useful information.

Sunday, 11 July 2010

I thought I would write about my experiences of using Mercury Quality Center (MCQ) to help manage my exploratory testing sessions

When carrying out exploratory testing I use the James and Jon Bach approach of Session based testing (http://www.satisfice.com/sbtm/). What I found is that the tool provided did not match the needs of the company and was hard to sell to management since we already had commercial tools for capturing testing effort (MQC). I had to re-think how I could get buy-in from management on using the exploratory testing approach whilst making use of the tools we already had.

One of the first things I did was implement a structure within the test plan section of MCQ. So I defined the following folder structure for each project

Project Name

Test Charter

Mission Statement

Test Ideas

e.g.

Project Name -->

Test Charter -->

Mission Statement -->

Test ideas(s)

So under the planning section testers can define a folder name for the test charter they are working on and then add a folder for each mission statement and then add their test ideas.

The thinking behind this was at a glance anyone can see what has been covered under each test charter and see if their any gaps. Reports can be pulled off and used during debrief sections to act as focus points when discussing the testing that has been done.

I created a Test Plan Hierarchy using a standard numbering scheme for the folder and test idea names. This helped with traceability and navigation around the test plan.

e.g.

Project Name -

01 – Test Charter 01 -->

01.01 – Mission Statement 01

01.01.01 – Test Idea 01

01.01.02 – Test Idea 02

01.02 – Mission Statement 02

01.02.01 – Test Idea 01

01.02.02 – Test Idea 02

02 – Test Charter 02 -->

02.01 – Mission Statement 01

02.02.01 – Test Idea 01

02.02.02 – Test Idea 02

MQC is setup for a formal test case and test step scripted form of testing, I have not found a way to get around this however instead of test cases I use test ideas and needed a quick way to create new test ideas without being bogged down in writing details about lots of steps. So I suggested that each test idea has ONLY the following information:

Test Idea Name

Test Idea description (This should be as descriptive as possible – include any models/heuristic thinking/problem solving ideas)

A single test step - This is required by MQC so that the user can run the test and record its status (Pass/Fail etc)

Since we use a different system for capturing defects (Don’t ask!) I also added a folder to each project called 99- Defects – so that I could trap any defects that needed testing.

The next step was to have a structure for the test lab (this is where details of tests are run)

I implemented the following structure:

Project Name -->

Project Release Version X.Y -->

01.01 - Mission Statement 01 -->

01.01.01 - Test idea 001

01.01.02 - Test case 002

01.02 - Mission Statement 02 -->

01.02.01 - Test idea 001

01.02.02 - Test case 002

It is recommended that X.Y numbers in Project Release Version name are provided as a multiple digit left zero padded integers. This is to ease sorting by name. This was basically copied over from the test plan section.

For exploratory testing I suggested that as a minimum the following columns are included when recording the execution of the test idea.

Plan.Test Name

Result

Defect (For recording CQ defects raised within that test script)

Priority * (How important is this test idea , what risk is it to the project by not doing this test idea)

Status

Execution Date

Once this had been setup it was then easy to run a session based upon a mission statement for the session I was running. Each mission statement had multiple test ideas. I found this very useful since it was very quick to create test focus areas based upon test charter names and mission statements. These could then very simply be turned into session sheets within MQC test lab.

One of the key elements of session based testing is to capture what all the evidence of the exploratory testing session. I implemented the following to capture details of what went on the testing session. Each test idea was run from within MQC and recorded if that test idea passed or failed. (I am aware this can be very subjective and depends on context however to ease transient to ET it is necessary to have some familiar ways of recording progress). I ensured that all session notes, log captures, screen prints, videos etc were captured by attaching them to the test idea.

THIS was very IMPORTANT – since if anyone needs to follow your test idea in the future they now have a record of what and HOW you executed your test idea. This is an issue with biases here and people carrying out testing afterwards could just follow your notes and repeat what you did which is not really exploratory testing but that can be mitigated by mentoring.

You now have a tool in which you can capture what you have done during your exploratory testing sessions.

There are a few issues I find with MQC and I am sure people out there in the testing community may have the answers. I want to use MQC to record the time spent on each session (As short, medium, long). I also I wanted to capture how much as a percentage of that time was spent on:

Test execution

Bug reporting and investigation

Test environment set up

Test data setup

This would help in the telling of the story of what is stopping the testers actually testing. I am sure there is a way to do this is MQC and I just need to do some more investigation. I hope readers of this find it useful, I know it has helped me to persuaded management to take exploratory testing seriously.

To finish this is working for me, it is not perfect and I am investigating other ways/tools that can make this more efficient. Looking at using a java application to create the session sheets and report back via the MQC API directly – but that is in the future. I am also investigating ways to customize MQC so that I can have the columns I wish to have. I will let you know it that works.

Friday, 25 June 2010

I read an interesting blog by James Christine yesterday (24-06-2010) (http://clarotesting.wordpress.com/2010/06/23/challenging-the-culture/) in which organizations which promote a positive/good news culture could be doing themselves harm by trying to encourage people not to report any bad news and how dangerous this is. I loved the alternative take on this and it got me thinking about a blog post I intended to do about how testers are perceived as the bearers of bad news and how we could change this perception. The article by James has spurred me to put the blog together.

I try to have a good working relationship with software engineers/developers/programmers (still struggling with what these highly talented people want to be known as) since most of the time I go and speak to them it is to say something is not working or it has crashed.

I had an eureka moment one day and took a step back to see why the relationship between the testing team and the software development teams were fragile and highly strained. I put myself in the shoes of the development team and how they perceived the testing team I asked the team how they felt about the testers and the responses I received all seemed to have a common ground.

‘I get a feeling of here we go again whenever a testers phones or comes to see me’

‘I dread it when a tester comes to see me’

‘They are always complaining that something does not work’

‘They only phone me when something goes wrong’

It got me wondering about how as testers we could improve this critical relationship and form a much better relationship. I thought that if everyday the same people are visiting/phoning me and giving me bad news I would soon develop a negative perception of those people.

So what can be done to change this?

I decided to try something a little different – someone once said from small acorns mighty oaks grow. I decided that instead of phoning or visiting the software development team when I had a problem I found the time to go and visit and ask how their weekend was or how the family is or what they thought of such and such in the news just a general chit-chat. One important thing I made sure I never did was to start to talk about general things and then say ‘Oh by the way … such and such does not work’ I cannot emphasize enough NOT to do this. My reasoning behind this was to build up a relationship and stop the feeling of dread when I turned up that something was wrong again.

The effect of this was amazing, the development team soon started to say hey have you seen this we are working on and start to talk with a passion about what they were working on. From a testing viewpoint this is valuable knowledge gathering. The attitude of the development team changed, when I did contact the team with a problem or something was wrong they would listen, emphasize and take a real interest in the problem rather than just dismiss it. The relationship between the teams improved tremendously.

So to conclude as a tester you do not need to always be the bearer of bad news to the development team. Take an interest in them as a person, take an interest in their lives and what they enjoy, take the time to learn about the people you work with. The benefits could be outstanding.

A word of caution on this – it has to be genuine – you really do need to be interested when you are talking to people about their personal lives. Otherwise you will come across as being cynical and shallow. If this happens then I am afraid you will cause an even bigger resentment and maybe even hatred of you.

Thursday, 24 June 2010

I recently ran an exploratory testing workshop in India and I thought I would blog about this experience.

There are many differing views about Indian software development teams, some which are unfounded and some that are characteristics of the working style of Indian teams.

From my experience of working with many different teams around the world the statement above can really be applied to any team no matter where they are in the world how people interact and their style of working is dependent on their culture and way of working.

The workshop I was running had a lot of interaction and required engagement from those attending otherwise it is hard to gauge if the audience understands what you are trying to deliver. I was worried due to what I had been informed about the culture within India that there would be little if any engagement and everyone would agree with what I was saying, even if what I was saying was wrong. (I like to set little traps in my presentations and say things which anyone in testing will know is stupid and start a debate.)

So taking on board the lessons Jon had learnt I started to change my presentation a little to become more personal more about who I was rather than what I was trying to deliver.

I changed the start of the presentation and included a lot of personal information about myself including photos which I had of my family. When I started to run the workshop I explained that this was just an approach, a possible way of working I was not going to say to anyone attending that this is how you MUST do things and if you disagree with anything I am saying then please let me know. I then spent the next 20-30 minutes explaining about myself and my family. I think that this part was the key element – the need to reach out to the India team on a personal level, family in India culture is very important (Joint Family). I talked about our daughter and granddaughter coming to live with us when her husband was away for six months with the army and lots more. I then asked people attending to talk about themselves and their families. Suddenly the atmosphere in the room changed it become more relaxed and people appeared more receptive.

Can this one little change make such a difference?

So I begin delivering the workshop and found the engagement and interaction of those attending to be amongst the best I have come across whilst I have been delivering this workshop. There was passion, interaction, thoughtful questions and in some cases surprising answers. In my opinion it was one the best workshops I have ever been involved with.

Was I just lucky?

If I had not changed the start of the presentation would I have still ended up with the same reaction and interaction?

I cannot really say since I have no comparison – this was a one chance to deliver to a team in India, I was on a tight schedule, so it was important to get it right.

I really must say a big thank you to Jon Bach, since without reading about his experience I think I would have blindly gone and presented and not got the response I required nor would anyone have really learnt anything.

So the tip for anyone working or dealing with teams from India is to make sure you can engage with them on a personal level , open up and let people know who you really are.

Thursday, 10 June 2010

I think I may have been a little remiss over the past couple of month by not updating my blog as much as I should be. I see posts by other great bloggers appearing every week or two but mine appear about once a month. I thought I would take time away from testing issues and blog the reason why I have not been as actively involved in the testing community as I would like to have been. This is a subject close to my heart and some may read and feel it is a little self indulgent however the cause IMO is more than worthwhile.

On Saturday 5th of June 2010 my wife and I organized in conjunction with the Amateur Poker Players League Europe (APPLE)

Apart from running the main poker tournament we had a pool tournament, a raffle and various other fun events. The support of local business was outstanding and overwhelming they could not do enough to help and given the current economic climate it was extremely humbling It was a different story with the large companies who I shall not name here who were not interested at all so when you think you need to pop out to get a pint of milk or buy something try to think of your local community businesses first rather than the big uninterested corporations.

The reasoning behind this is that our son-in-law (Lance Corporal Matthew Wellington) who is in the Royal Engineers returned from his tour of Afghanistan. His role with the Royal Engineers is with the EOD (Explosive Ordnance Disposal,) which as you can imagine is a highly dangerous and stressful job. He has a daughter who is now 2 years old and unfortunately has only seen her daddy for about 1 year of her life since this is Matthews’s second six month tour of Afghanistan within two years.

He has done his duty whilst the family at home, apart from the natural worry, felt helpless, so organized this day to help provide something back to those who are serving and the unfortunate ones who return injured. During his current tour he had to go through the trauma of losing some colleagues and a few who came back suffering from horrific injuries.

So you can imagine my wife Tracy and I had lots to organize and do, which took our minds away from the worry of our son-in-law whilst he was on tour, dreading watching the news and of hearing another member of the armed forces had been injured or killed. It has been a very stressful time and to be able to do something good has helped a great deal. At the end of the day the final amount we raised for this cause was over £1500.00 not bad for a single day event.

Part of this blog is to raise awareness of Help for Heroes and all that they do. They are not politically motivated and are doing a wonderful job and ensuring members of the UK armed forces are rehabilitated in an environment suitable for such heroes. SO if nothing else after reading this blog please visit the Help for Heroes website and maybe just maybe make a small donation.

Monday, 17 May 2010

I thought I would write a brief blog about a meet up of testers whilst I was in Bangalore, India.

It started with me sending a tweet out asking if any testers in Bangalore were interested in meeting up at the weekend. After a short while Santhosh Tuppard (@santhoshst ) responded by putting an invite on his blog site (http://tuppad.com/blog/2010/05/13/bangalore-testers-meetup-–-may-15th-2010/) then all of a sudden there was a flurry of activity on twitter as people asked about timing and where and when.

So on the Saturday I made my way down to the forum and met up with seven testers from Bangalore.

+ two others – if you can remember who they add please add their details as a comment.

We had an agenda or some points to talk about:

Should we record our emotions when testing?

Exploratory Testing – good and bad points – how can we solve the bad parts?

Measuring Testing quality – How?

If we kept to these themes was a different matter.

After some brief introductions everyone was keen to know about me. After explaining I had been in IT for over 24 years which was older than some attending!!! Pradeep pointed out I did not look that old!!!!

We then started to discuss some testing issues and the following is what I remember of the conversations that took place. I am sure some of the others who attended will correct any details I have got wrong :o)

One testing issue we looked at is when teams are being managed remotely and managers at not in the same location as the testing team. Pradeep says this becomes a problem because managers want a breakdown of the testing activity all the time and the testing team spend a great deal of time answering questions from managers about what they are testing.

My thoughts on this was that anything that is actually stopping a tester from testing should be recorded and reported so at the end of each test phase/cycle when reporting back to management if a lot of time is being spent reporting test activities to managers rather than testing then this can be clearly seen and a ‘good’ manager will try to rectify it. One idea suggested was that the managers should become more hands on and take part in the testing activities. The exploratory testing approach makes this very easy to do.

There was a lot of interest in reporting what is stopping testing rather than any thing else – this lead on to a discussion of measuring testing and testing quality. This is always a difficult question and I am not sure we fully answered it in our discussions. I mentioned that when testing should we record our feelings and emotions about what we are testing. I suggested that we use a happy, sad face system so when we feed sad about testing we have a sad face when we feel angry an angry face and so on, It would then be very easy to see a trend for a area under test. If there were lots of angry faces then someone will spot and trend and start asking questions of the testers about why they feel this way. Pradeep pointed out that developers may try to influence the testers to always make happy faces. I stated that testers should be still independent of developers and that we should support and help developers but we would not tell them how to write an algorithm so why should they tell us how to test?

On the subject of measuring testing the discussion revolved around looking for trends rather than looking at numbers. The numbers could still be wrong but that is where talking helps to understand what the numbers and trends are saying.

Another point was made about using Tech support people as a resource for testers. I posted this on twitter as a question and got the following responses: (may not be in time/date order)

testingqa: @steveo1967 Tech support people hold some of the skills, but I wouldn't immediately say it implies they will be a good tester too #testing

testingqa @QualityFrog @michaelbolton My 1st role in IT was Tech Support role,those I worked alongside were slack & passed along cases for me to solve

testingqa @QualityFrog @michaelbolton So my faith in Tech Support staff isn't great ;) But I do agree that a good tech support person can hold an... appropriate background useful for testing, but even then Tech Support doesn't require an eye for detail often.

qualityfrog @michaelbolton @testingqa And, sadly, some in tech support are about as astute at faking support as some testers are in faking testing.

michaelbolton @testingqa Not automatic, but the skills and experience of support greatly overlap those for #testing. Great foundation, I'd argue.

michaelbolton @QualityFrog @testingqa I took *good* tech support person as read. Your pessimistic interpretation is also valid, alas.

So even here there are two sides to the argument but it appears that ‘good’ tech support could be valuable resource to testing. Maybe I am a little biased on this since my background is technical support and I believe that if you have a good grounding of how users think and technical knowledge as well (preferably at a coding level) then you have the right attributes to look at becoming a tester.

It was interesting that whilst talking to the group and I was aware I was doing a lot of the talking how much interest was shown in the subjects we were discussing, more importantly how much passion all the people at the meeting had for testing. I could see that in everyone’s faces a passion for learning and understanding. Key attributes of ‘excellent’ testers. I twittered about meeting the future teachers of software testing and I still believe that these people hold the key to the next 10 – 20 years of software testing.

The next subject that came up was one of certification – I explained that on the way to the office the other day I saw a sign that said “Learn software testing in two weeks”, everyone in the group laughed. We talked about the fact that software testing is not like programming in which once you have learnt the foundations of the language you then have the building blocks to create code. Software testing is not computer science but a mixture of many types of science including psychology and philosophy. It is not an exact science and there is plenty of scope to get it very wrong which we have all seen. The question of certification seems to be a very interesting one considering the latest blog posts coming from James Bach and Stewart Reid. (http://www.satisfice.com/blog/archives/457, http://www.eurostarconferences.com/conferences/session-details.aspx?sessionId=209)

I do not understand the need why fellow professionals need to attack each others ideas, it is getting very political. This is just my own personal viewpoint and nothing to do with the meet up. I believe in the need to debate and discuss ideas and opinions but when it appears that someone is personally attacking someone it then appears like a personality problem and distracts from a meaningful debate on the issues – I really want to attend Eurostar now just to hear the debates.

This nicely leads us to the subject of certification that was brought up during the meet up. We discussed certification and all of us agreed that it appeared to be a money making scheme that gave no benefit to experienced testers. The concern of the group was that agencies would now demand that testers have this certification and people will take the exam worried that if they did not they would not get a job in testing. The group felt this was very wrong, to be pressurised in taking an exam because there is no other measurable way to prove you are an ‘excellent’ testers. As a group we came up with some ideas that may be a way forward and needs everyone in the testing community to push forward.

Should try to meet fellow testers a couple of times a year at testing meet up – the internet allows this very easily as this meet up has proved

Once this is done when you apply for a job and the agency asks for your certification – point to your active online presence in testing ask them to talk to peers who have met you and who can vouch for you. If we ALL did this then the need to pay organisations to prove you can test goes away. Let us as a testing community certify each other.

So with this last thought I did a little experiment on the people in the meet up involving the calculator experiment as taught by Michael Bolton – if you have experienced this from Michael then you will know what I mean if not then you need to find someone who knows because I am not going to spoil it by explaining on here. Thank you again Michael for giving me a useful way to demonstrate a key point.

So all too quickly the meet up had to end. Did we all learn something? I hope so. I know I did and I had a wonderful time and left feeling encouraged and motivated. I still think I talked far too much and for that I apologise hopefully if there is a next time I will encourage others to speak even more – you have been warned……