Friday, July 27, 2007

"No device has been invented which can match the human eye for accuracy and precision in many tasks; nor the human ear. Even the admittedly handicapped human is able to identify and distinguish among far more scents than any automated device. Human pattern recognition, language recognition, and language production ability is well beyond anything currently envisioned by automation engineers." - Answers.com

Hope you have heard many times the word “Automated Testing” (Some places I even have heard “automatic Testing”)

Have you ever thought – what does that mean? Let us dig little and try to figure out what each term in above phrase *might* mean?

We have two words in above phrase – “Automated” or “Automatic” and “Testing”. Let us explore possible meanings of these terms ….

(A) Automated – something that happens without any human intervention – something that is done by a machine-software program or by self.

Answers.com says -"automated" (adj )Definition: made or done by a machineAntonyms: by hand, manual

The word testing has many meanings – I present two contrasting definitions

(B) Testing is an act of technical investigation performed on behalf of stakeholders in order to reveal quality related information – it is an act of questioning – An infinite search for problems – problems that can annoy/irritate/destroy/damage a stakeholder.

(C) Testing an act of executing a test case derived from a specification and verifies whether the test case passes or fails.

(D) It is an act of confirmation that software behaves in a way as prescribed by the underlying specification/design.

(E) Testing proves that the software works as desired.

Now think about constructing the meaning of the word “Automated Testing” and we have these definitions (A), (B), (C), (D), (E) …

Automated Testing = (A) + (C) – Seems possible – but in a narrow way. Using another program (automation tool) one can execute an automated test … but that is only a part of whole story – what about Test design? Defect investigation? And host of other activities that fall under testing umbrella …

Automated Testing = (A) + (D)Automated Testing = (A) + (E)

Both of above seem to be “Not possible” as checking or proving that software works the way intended or as per spec (writing in English – hence can be interpreted in infinite number of ways) using “automation” – solely – is not practically possible.

So … think twice before using the term “Automated Testing” – as nothing really EXISTS in this world of testing called as such …

Challenge me … think of ways where you can justify the use of “automated testing” – come up with definitions of “automation” and “testing” and link them !!!

"Insanity is doing the same thing, over and over again, but expecting different results"Albert Einstein (attributed to Ben Franklin too)

Does this sound similar to what we do in the name of "regression testing”? In regression testing (more so in that regression testing whose test execution is automated - note I am not using the term "automated regression testing") , we believe that by doing repetitions of some things that we did previously - we can find new results.

Thanks to inherent complexity in software and involvement of humans in development, testing and usage of the same - this regression insanity appears to be *working*

This one of those Francis Bacon's idols - Idols of the Theater: Errors formed from dogma (institutionalized doctrine) and flawed demonstrations.

Read following to know more about Francis Bacon and how his idols are relevant in testing

The boundary condition of the universe is that it has no boundary." The universe would be completely self-contained and not affected by anything outside itself. It would neither be created nor destroyed. It would just BE

Stephen W. Hawking - A BRIEF HISTORY OF TIME

Science is a differential equation. Religion is a boundary condition- Alan Turing

Boundaries are actually the main factor in space, just as the present, another boundary, is the main factor in time- Eduardo Chillida

Earth has its boundaries, but human stupidity is limitless – Gustave Flaubert

I would read above quote from French novelist of 18th century (Flaubert was regarded as the prime mover of the realist school of French literature and best known for his masterpiece) by replacing the word “Stupidity” with “intelligence” or “imaginative power”.

Understanding, Identifying and working with boundaries in software - is a Modeling problem. There can be multiple ways in which boundaries in a software system can be modeled.

Depending upon how *close* your model models the system behavior, your (boundary) testing can be incomplete or wrong to *that* extent. Since there can be boundaries out side your model – you will not notice them.

There can be (are) multiple boundaries - what you notice is limited by your model (ref. #1)

No boundary exists in isolation

All boundaries, those that are explicitly identified in model – hence are known to you and those laying outside your model, INTEREACT and ALTER/AFFECT system behavior in some *significant*. This introduces complexity in the model. In other words there exists some RELATIONSHIP between these boundaries.

Inputs in identifying (in fact for modeling) the boundaries come from various sources such as technical specifications, user expectations, requirement specifications, device/OS specifications. If you narrowly focus on any one source of information – you will miss modeling of others boundaries (outside the mode) hence you would miss bugs/system behaviors associated with them.

A weak boundary Analysis would start with boundary first without first having a thought /explicitly/ about my model.

A strong or useful boundary analysis would start with model and identifying boundaries resulting from all possible sources of information.

Don’t forget to check following ---

All models (those especially having predominant influence of domain testing) are APPEAR to be deterministic (algorithmic/mathematical representations - set, graph, state machines etc) but indeed are HEURISTIC models as act of modeling seem to follow (in most of the cases) a heuristic approach…