When it comes to actually carrying out test execution and by that I mainly mean manual testing in an exploratory way by the use of session based test management. There is a need to have a fairly even split between critical and creative thinking. When you are doing test execution you are carrying out

When in the test execution phase the spilt between the two styles of thinking is evenly match there are times when you will need to think creatively and time when you need to think critically. Switching between these different styles is not easy and it is a hard skill for a tester to master. The more you practice the easier it does become. However session based test management can be very powerful to help with your thinking styles since it encourages the use of focused and uninterrupted periods of time which is vital for both styles of thinking to be fully utilised. To do this correctly it may be a good idea when doing test execution to do the following:

Switch off email

Unplug phone

Put up a sign saying “please do not disturb thinking testing in progress.”

Carrying these simple things out may help to remain focused on your thinking process when testing.

It is not possible to do two thinking tasks at the same time not matter what you believe you think you can do. See here, here and here.

So you need to be creative in thinking what you may want to test next based upon what you are experiences whilst testing. At the same time you are learning about the system and need to think critically about what assumptions you are coming up with from what you see and deciding if what you are doing is the best use of your time.

One method that can be used when testing is to look at test framing as a critical thinking approach that can be useful during the test execution phase. It provides you some questions that can engage your critical thinking side such as:

Why are you running (did you run, will you run) this test (and not some other test)?

Why are you running that test now (did you run that test then, will you run that test later)?

Why are you testing (did you test, will you test) for this requirement, rather than that requirement?

How are you testing (did you test, well you test) for this requirement?

How does the configuration you used in your tests relate to the real-world configuration of the product?

How does your test result relate to your test design?

Was the mission related to risk? How does this test relate to that risk?

How does this test relate to other tests you might have chosen?

Are you qualified (were you qualified, can you become qualified) to test this?

Why do you think that is (was, would be) a problem?

It should be noted that these questions can be used to during any of the testing stages. I have found that test framing from my own experiences provides the most value during the test execution phases since the problem you are trying to frame is at the forefront of your thinking.

When carrying out your testing session it is important to think critical about the priority. This is something that we tend not to think too much about when testing and mainly think about this during planning or when reporting bugs. I feel when you are in the test execution phase it is important to use the framing techniques mentioned above to adjust your priorities based upon what you are uncovering and the new information you are being presented with. It is not a race to complete your testing sessions and slowing down just a little to think critically about the testing you are doing and thinking about the value of what you are doing at that moment and adjusting the priority based upon your thinking. There are dangers within this approach in that your thinking could be clouded by your many biases , however this can be mitigated later during the test analysis phase.

On the creative side it is important to remember that one of the key parts of exploratory testing is to look for new opportunities to test and the best way to do this is to thinking creatively and as you uncover new stuff that you had not thought about testing before to make a note of this. Note taking is important for effective testing. If you do not make notes you will have a strong chance of forgetting that great idea or concept you had thought about at the time to test. It is important to capture your thoughts and your ideas for future use. This can be done in many ways from the creative use of video , capture and replay tools, wikis , mind maps , session recording tools (Session testerRapid reporter) to good old pen and paper . If using pen and paper a good way to make your creative ideas stick is the use of drawing and images, I highly recommend the book ‘The back of the Napkin’ by Dan Roam for more information on this. I also suggest talking to Andy Glover (78) (@cartoontester) since he has a lot of great thoughts and ideas on the use of creativity in testing.

How you capture is not important, it is what you capture that is important. One issue I have had over the years of doing exploratory testing is how much is enough details for my notes. One creative method I have found useful is at the end of each day I email my session notes to myself and the first email the next morning I read is my session notes. If I cannot understand them at that time, less than 24 hours after I wrote them, how can I expect other people to understand them? This has over the years enabled me to refine the amount of details I have in my session notes and become better at note taking.

As human beings we seem to be very afraid of failure and try to avoid being in situation in which we are wrong in our thoughts and ideas. In my opinion the best part of using exploratory testing is the ability to be able to test lots of your thoughts and ideas which you have just thought about when testing and not being too bothered if your ideas are right or wrong. It is about proving your theories and it costs very little if they prove to be wrong. We need to have and test lots and lots of our ideas to help us understand more about the system in doing this we are going to lots of failures of our ideas and in many cases we are going to be proved wrong.

I have wrote previously about placing many bets and how useful this is for testing. Franz Johansson in his book ‘The Click Moment’ explains more about the theory being this thinking. To be creative we need to embrace failure and see the learning opportunities this provides us. There is an interesting article on this concept of failure helping us to learn more which has been given the title of ‘Productive failure’. It appears that we can learn more and become better by being allowed to fail rather than only being correct. There are other articles on the fact that allowing us to be confused is good and encourages us to learn more. So when you are testing it might be good sometimes to be confused about the software since it may help you learn more.

So to summarise the execution phase as a tester you need to use both styles of thinking equally. You need to be creative in how you test and what ideas you come up with to test at that time and in the future. Alongside this you need to think critically about what you are currently testing and is what you are doing right there and then the right thing and do your biases influence your approach. It is not easy being to switch between these two styles of thinking during a period of testing and maybe this explains why some people when testing feel mentally exhausted after a testing session since there brain have been engaged in a constant battle between creative and critical thinking.

The next section will look at the style of thinking needed to analyse the testing that has been done and the evidence you have gathered.

Test Planning

The traditional approach to test planning has involved mapping requirements to test cases and creating test scripts with steps and expected results. This activity has started to be replaced with the use of automation especially using frameworks such as cucumber and behaviour driven development methods. In conjunction with this testers use the exploratory testing approach of creating charters and missions to create test ideas. During test planning testers should be primary using creative thinking with some critical thinking.

Since the cognitive outcomes are dependent on thinking of ideas, new things/ways to test then the best way to achieve this is by the use of creative thinking. Testing is very good at finding ways in which the software does not work and our role is to exercise the software the best we can and by planning at a high level novel and interesting ways to do this based upon our skills, knowledge and experience is a good exercise in creative thinking.

There is some overlap here with test execution since when we test we find more to test and whilst following our mission find more creative things to do.

So to encourage our creative thinking we need to use tools and approaches that are more suited to this way of thinking.

Mind Maps

One way to do this is to not have too much structure and allow the person to be able to follow their creative processes. The use of mind maps allows people to capture their thoughts and ideas. They then can amend, group, ungroup, change the information they have thought about in a way that encourages and follows their creative thinking process. It provides a visual reference of their ideas and allows them to see a bigger picture. There are many articles here, here and here on the use of mind maps to aid creative thinking but to quote from one

Currently the majority of teams, I have been involved with, are working with Freemind as the de-facto tool for mind mapping within the organisation. There are a great many articles on the use of mind maps for test planning and for those interested you can read more here, here and here.

During the planning stage there are occasions when you suddenly become blank and cannot think of new things that you plan to test. This happens in all creative activities (commonly known as writers block) however there are tools and techniques that can help to overcome and that is what we shall look at next.

Checklists and cheat sheets

One thing that can help generate ideas is the use of checklists and cheat sheets. There are many examples of these within testing and a few that can be recommended are:

Brainstorming

There are arguments for and against using brainstorming for improving the creative process. However the research appears to side with the fact that collaboration with others can has a positive effect on creativity as long as the people involved encourage and support rather than criticize and say negative things. It has to be handled in a sensitive way for collaboration to work when thinking creatively. It is important for creative to work correctly to be in a creative environment and there are many articles giving suggestion on how this can be achieved (Creating a creative environment for braining storming), (Creating innovate workplaces). The benefits are that another person’s perspective can trigger a new path to follow or spot something you may have missed or does not make sense.

If you are asked to review someone’s creative work and ideas you have a vital role to ensure you do not become too negative and support and encourage their ideas. It is very easy to squash someone creative side by using negative throwaway comments. I am not saying there is no need for debate and discussion but it needs to be discussed with empathy and respect.

There are some interesting articles about feedback and they can be found here, here and here.

Test Tours and Personas

Within testing there is a concept of testing tours which have been around for quite a while and they can prove useful for helping to drive creativity by placing . They are many useful tours which can help you think about a certain function or feature and find novel ways to test it. Some examples include:

Documentation Tour: Look in the on-line help or user manual and find some instructions about how to perform some interesting activity. Do those actions. Improvise from them.

Feature tour: Move through the application and get familiar with all the controls and features you come across.

Testability tour: Find all the features you can use as testability features and/or identify tools you have available that you can use to help in your testing.

Continuous Use: While testing, do not reset the system. Leave windows and files open. Let disk and memory usage mount. You’re hoping that the system ties itself in knots over time.

There are many more and I have provide some links to them here , here, here and here. These can be used during the planning stages to help you focus on certain areas and with creative thinking devise novel ways to ‘tour’ around the application

Testing personas are fictional people who represent a typical type/group of users who will use the software. When you are thinking creativity you can come up with many different types or groups of people and then note some ideas of how that persona would test the software. Similar to testing tours there have been many articles on using and creating testing personas. Some of which I have added here:

There are unlimited personas that can be created and you are only limited by your imagination and hopeful these references will provide you with enough information to create your own.

The “what if” question

One find way to help generate ideas for testing is to ask yourself questions in which you do not know the answer but think there would be benefit in finding out the answer.

For example:

What if I tried this?

I wonder what would happen if I did this?

What if there was no (add your own word here)?

How many ways can I do this?

Can I make it do the opposite?

Is there anything that I can change that may affect the software?

These are good starting points if you feel you are running out of ideas and can help create opportunities for more avenues to explore.

Automation

One part of the test planning is to see what information you already know about and look at ways to automate this using BDD or any other method that works for you. You need to use critical thinking to see which of your creative ideas would be most suitable to automate and look at creating a list of these possible automation opportunities.

Critical thinking of creative ideas

One important note to make about creative thinking is that once you have created your ideas you must then use some critical thinking to ensure that your ideas are sound. As Ken Robinson points out in his book “Out of our minds”

“Creativity is not only about generating ideas; it involves making judgements about them. The process includes elaborating on the initial ideas, testing and refining them and even rejecting them, in favour of others that emerge during the process.” (Our of Our Minds - Ken Robinson)

The next article will look at test execution and the thinking style that might be best for that.

*Oops - Had to do some editing of the article and add some references that were missing.

Documentation review

Even though we are moving towards a more ‘Agile’ style of software development it does not mean here should be no documentation and for complex projects it can be vital. There are various documents that can be created and the ones most commonly accessed by testers are the requirements, design specifications and high level design. The old schoolers may remember this being referred to as ‘Static Testing’, 'Inspections and walkthroughs.

One of the early stages of testing is for testers to review requirement documents from a testability perspective. When reviewing the document the tester should be asking questions about the statements been made and critical thinking about what has been stated. This stage of testing for the tester should mainly be a critical thinking exercise with some aspects of creative thinking.

The question is how do we apply critical thinking to documentation review?

One way is by the use of check lists and heuristics (rules of thumb) to prompt our thoughts on the testability of the requirements. One example of this is shown below:

Do questions need to be asked of the requirement before a test can be derived? If so, it is incomplete.

Are there two or more requirements that are in conflict with each other? If yes, they are inconsistent.

Can the requirement be interpreted in more than one way? If it can it is ambiguous.

Does the requirement contain the words AND, OR, NOR, IF-THEN-ELSE? If it does, it is likely to be non-compound.

Does the requirement fail to comply with any of the five criteria? If yes, then it is not testable.

Does the requirement deal with the ‘what’ rather than the ‘how’ (e.g. design)?

For example, the requirement ‘Provide a database’ states how a function should be implemented rather than what it is that is required. It should instead read ‘Provide the capability for traceability between requirements’

Is the requirement written as an operation rather than a requirement?

For example, ‘It will be stored in a x20 rack’ describes the operation rather than the environment. It should read ‘It shall be designed for storage in a x20 rack’.

Is the requirement written using the correct terms?

For example, the terms shall, will and should:Requirements use shallStatements use willGoals use should

Does the requirement contain the words is, was or must? These should be removed.

Are there any requirements that are missing? (e.g. reliability, quality)

Other sources/check lists that can be used to encourage critical thinking of the document under review include:

There are some similarities between the heuristics and models you use when following test execution and when you are reviewing documentation since your mind is trying to flow through how the program with function so some of the model you use for test execution can also apply to reviewing test documentation.

For example you can use the consistency heuristics created by Michael Bolton and James Bach. HICCUPPS

Inference: What are my assumptions about the requirements? Are my assumptions correct?

Conference: speak to the rest of the implementation team about the issues around testing.

I am not suggesting using all of these methods but to mix and match and choose which one works best for you. You may choose not to use any of them and create your own (this is the creative thinking element) If you do create your own it would be great to share with others.

So let us try to apply this in practice.

Incomplete example:

The system shall restrict access’
Should be rewritten as:
‘The system shall control access via usernames and passwords’

Consistency example:

‘The system shall calculate all distances in miles’
‘
The system shall calculate all speeds in km per hour'
These should be rewritten as:
‘The system shall calculate all distances in miles’
‘The system shall calculate all speeds in miles per hour’

‘An employee is entitled to 4 weeks holiday a year’.
Should be redefined as:
‘An employee is entitled to 20 working days holiday per calendar year

Embedded (compounded) example:

‘The target platform for the server system shall be Windows 2000 and Windows NT for the client system’
Should be separated into 2 requirements:
‘The target platform for the server shall be Windows 2000’
‘The target platform for the client shall be Windows NT4’

A good, well-thought-out product specification, with "all its t's crossed and its i's dotted," has eight important attributes:

Complete. Is anything missing or forgotten? Is it thorough? Does it include everything necessary to make it stand alone?

Accurate. Is the proposed solution correct? Does it properly define the goal? Are there any errors?

Precise, Unambiguous, and Clear. Is the description exact and not vague? Is there a single interpretation? Is it easy to read and understand?

Consistent. Is the description of the feature written so that it doesn't conflict with itself or other items in the specification?

Relevant. Is the statement necessary to specify the feature? Is it extra information that should be left out? Is the feature traceable to an original customer need?

Feasible. Can the feature be implemented with the available personnel, tools, and resources within the specified budget and schedule?

Code-free. Does the specification stick with defining the product and not the underlying software design, architecture, and code?

Testable. Can the feature be tested? Is enough information provided that a tester could create tests to verify its operation?

There are many other areas within the review phase in which critical thinking can play an important part which we have not touched on within this article and I suggest people reading this article to go and investigate more where testers can add value to a development project by thinking critical. We can influence such areas as code reviews, walkthroughs and inspections.

The next article will look in depth at Test Planning and the style of thinking required for that stage.

Monday, 11 March 2013

The last blog post looked at what critical and creative thinking was and which may be most useful during different stages of testing. This post will start by giving an overview of each stage and then look in depth at each of the stages.

Stages of testing.

Before we start to look at how we can use these types of thinking within testing we need to breakdown the testing life cycle into manageable stages. This is not an expansive list of the many stages but the labels given are to give a high level overall view and simplify the classification for which style of thinking is most suited for each phase. It not meant to state that these are the only things we need to thing about in stages nor does it mean that there is not any cross over between stages. They are meant as ‘guide’ lines only and not as best practice, process or otherwise.

Documentation Review (Static Testing)

Requirements

High Level Design (HLD)

Feature documents

Specifications

Standards

Regulations

Code review

Walkthroughs

Inspections

Oracles

Documentation review does not just mean reviewing requirement documents it could mean reviewing code, database models, country laws and standards. Requirements are only ONE source of information (Oracle) Instead of going into great detail about what role testing plays in the documentation review stage there are many resources available on-line that can do a far better job than me. I highly recommend that people look at these links if they wish to know more about static testing and documentation review.

* Disclaimer - I may not agree with all the approaches; processes and methods being suggested in the above links but they provide information about the diversity of what documentation review is in the context of software testing

Test Planning

Test Ideas

Automation – feature files (cucumber)

Missions/Charters

Mind mapping

The test planning stage is one which is much maligned with people within the testing community. There are people making all sorts of statements about the purpose of test planning with some saying to do as little as possible (Agile style) with others saying it should be wrapped in best practice and process and standards. In my own world there is a important need for planning but there is a case that says do not do too much. I feel that with test planning you should do enough to enable you to start doing some actual testing. Once you start testing you can then update and maintain your test plan based upon what you experience and discover. I try to align myself to James Bach's approach to test planning which can be found here page 21.

Test Execution

Now we come to the key element of testing, the part where the tester should engage their brain and start to ask questions of the software. I am talking about manual testing and specifically exploratory testing. There are many articles on what is exploratory testing and how to do it and rather than repeat information I will provide just one link to the excellent resource by Michael Bolton here. This is one page I do have bookmarked and come back to time and time again.

Test Analysis

Bug investigation

Defect reporting

Repeatability

Questioning

Future Automation

Debrief

Future missions/Charters

This is the often forgotten about stage of testing in which the tester takes a little time to reflect on what they tested, did they do the best they could within the constraints they had? Could they have done things differently? This time could be spent reviewing the session notes and investigating issues they uncovered during the execution phase, that they thought could be bugs. It could be used to check if a bug was repeatable and then raised in a defect tracking system with full evidence of how to recreate. It could be looking at what may be useful to automate from what information was discovered. It could be adding new ideas from the notes for future testing. It is also the time to debrief and talk to others about what you did.

Test Reporting

Dashboard

Wiki – sessions

Updating plan

Qualitative

Quantitative

Test reporting is a highly controversial subject within the testing community with lots of discussion about the use of metrics. There are some great articles out there on this subject and I am not going to attempt to summaries it here. If you want to understand more about the metrics debate then can I suggest you start with this blog post of mine. Then have a look at some of the following articles from Michael Bolton's resources.

To me the best way to do test reporting is the telling of a story using numbers to help support the story. A good way to do this is by the use of dashboards, a great example of this can be found here.

How does this apply to testing?

So now we have our testing stages we can start to see which style of thinking would be most suited for this stage, not forgetting that the other style will and should still be used, but the focus should be on the primary style of thinking for that stage.

The next article in this series will look in depth at documentation review and the style of thinking that may be most suited for this phase

Tuesday, 5 March 2013

I was asked a question recently regarding which style of thinking to use when at different phases within testing, creative or critical and I started to think deeply about this. I quickly sketched a diagram to explain my thoughts to the person who posed the question. Then I thought I need to explain my thinking behind the diagram and started to write an explanation for each section of the diagram, this has turned into a rather long research paper and as such I thought I would share my finding on my blog. This will be a series of blogs since I think there could be too much information to digest in one reading.

I do not know if my thoughts and my concepts are correct but the best way to find out is to publish and await peer review. Thanks to @simtesting for spurring me to get this blog post started. So your comments, opinions and thoughts would be most welcome.

Where to start? The first thing would be to show the diagram that started it all off.
The diagram can be download here

Some of the terms I use for stages may be different to what you use but as this series of blogs goes along I will try to explain what I mean by them and also what the diagram represents.

This diagram shows the emphasis of thinking for each testing stage. The larger the area the more of that style of thinking should be used. To summarize when reviewing documentation you should be more focused towards critical thinking with a small amount of creative thinking. During the test planning stage this is reversed with more effort placed upon creative thinking rather than critical. When execution your testing there is a balance between being creative to find more stuff to test and thinking critical about what you experiences and the theories you are forming. After you execute your testing you need to be critical of what you have observed and see if there are any problems with the testing you carried out and see if the information you have captured holds up to critical analysis. Finally you need to report on your testing and you need to find creative ways to present your information without overloading the recipient and confusing them with meaningless data. To do this you should have a strong slant towards thinking creatively but also keeping in mind a critical viewpoint of what does that information really represent and how it may or may not be misinterpreted. Then repeat the cycle again and again!!

This series of blogs will look at each of these stages and give examples of how we might be able to improve the style of thinking required for that stage.

To start with it may be a good idea to define what critical and creative thinking are so that we can give context to the subject.

Defining Types of thinking:

The first thing that needs to be done is to look at what we mean by critical and creative thinking to ensure we all understand the same meanings.

“Critical thinking is the active, persistent, and careful consideration of beliefs or knowledge in light of evidence and creative thinking is the generation of new ideas. Critical and creative thinking are fundamental to human intellectual progress and artefacts thereof. Depending on context and purpose, critical and creative thinking skills can be interdependent or separately applied.” Creative and Critical Thinking Definitions

Or in simple terms it is a way of thinking that can help you decide if a piece of information is always true, sometimes true or partly true or false. It is a way of thinking which allows you to question the statement(s) being made.

There are main forms and practices of critical thinking.

One I personally use is

Recognize the argument – (ask is there a problem here?)

Analyse the argument – (If there is a problem, why is there a problem)

Many people see creative thinking as mainly artistic and that they are not creative thinkers. However being creative is more than art and it can apply to all areas of your work. The difficulty is being able to focus on innovation and treading a path that has not been mapped. A little bit like what we do when exploring. (Can you spot the connection yet?) .

Summary of the styles of thinking

We need to be mindful that critical thinking is important but we should not forget about creative thinking, there is a balance that needs to be made. Which has more influential benefit depends on the context and the stage you are at in testing. The rest of this series of blogs will look closely at each stage and offers some thoughts on which type of thinking may offer the most return.

The next blog article will look at defining the stages before looking closely at each stage in turn.

Update

Updated to include diagram as suggested by Michael Bolton in the comments

What do you think?
Which do they prefer?
The original one or this one?