Transcript of "Software Testing"

1.
Motivation
People are not perfect
We make errors in design and code
Goal of testing: given some code, uncover
Software Testing as many errors are possible
Important and expensive activity
Not unusual to spend 30-40% of total
project effort on testing
For critical systems (e.g. flight control): cost
can be several times the cost of all other
activities combined
CSE 757 - Software Testing 1 CSE 757 - Software Testing 2
A Way of Thinking Testing Objectives
Design and coding are creative Testing is a process of executing
software with the intent of finding
Testing is destructive
errors
The primary goal is to “break” the software
Good testing has a high probability of
Very often the same person does both
finding as-yet-undiscovered errors
coding and testing
Need “split personality”: when you start Successful testing discovers unknown
testing, become paranoid and malicious errors
Surprisingly hard to do: people don’t like If did not find any errors, need to ask
finding out that they made mistakes whether our testing approach is good
CSE 757 - Software Testing 3 CSE 757 - Software Testing 4
Basic Definitions Testing Approaches
Test case: specifies In this course: small sample of
Inputs + pre-test state of the software approaches for testing
Expected results (outputs and state)
White-box testing
Black-box testing: ignores the internal Control-flow-based testing
logic of the software Data-flow-based testing
Given this input, was the output correct?
Black-box testing
White-box testing: uses knowledge of Equivalence partitioning
the internal structure of the software
e.g. write tests to “cover” internal paths
CSE 757 - Software Testing 5 CSE 757 - Software Testing 6
1

3.
Statement Coverage Example
Given the control flow graph, define a Suppose that we write and
execute two test cases 1
“coverage target” and write test cases to
achieve it Test case #1: follows path 2
1-2-exit
Traditional target: statement coverage 3
Test case #2: 1-2-3-4-5-
Test cases that cover all nodes 7-8-2-3-4-5-7-8-2-exit
(loop twice, and both times T 4 F
Code that has not been executed during take the true branch)
testing is more likely to contain errors 5 6
Problem: node 6 is never 7
Often this is the “low-probability” code executed, so we don’t have
100% statement coverage
8
CSE 757 - Software Testing 13 CSE 757 - Software Testing 14
Branch Coverage Branch Coverage
Target: write test cases that cover all Statement coverage does not imply
branches of predicate nodes branch coverage
True and false branches of each IF
Example: if (c) then s;
The two branches corresponding to the
By executing only with c=true, we will achieve
condition of a loop
statement coverage, but not branch
All alternatives in a SWITCH coverage
In modern languages, branch coverage Motivation: experience shows that many
implies statement coverage errors occur in “decision making”
Plus, it subsumes statement coverage
CSE 757 - Software Testing 15 CSE 757 - Software Testing 16
Example Achieving Branch Coverage
Same example as Branch coverage: a necessary minimum
1
before: two test cases Pick a set of start-to-end paths that cover
Path 1-2-exit 2 all branches, and write tests cases to
Path 1-2-3-4-5-7-8-2-3- execute these paths
4-5-7-8-2-exit 3
Basic strategy
Problem: the “false” T 4 F Add a new path that covers at least one edge
branch of 4 is never 5 6
that is not covered by the current paths
taken - don’t have Sometimes the set of paths chosen with this
100% branch coverage 7 strategy is called the “basis set”
8 Good example in Pressman, Sect. 17.4.3
CSE 757 - Software Testing 17 CSE 757 - Software Testing 18
3

5.
Data-flow-based Testing Black-box Testing
All-definitions: for each definition, cover Unlike white-box testing: don’t use any
at least one DU pair for that definition knowledge about the internals of the code
i.e., if x is defined at n1, execute at least
Test cases are designed based on
one path n1…n2 such that x is in USE(n2) and
the path is clear of definitions of x specifications
Motivation: see the effects of using the Example: search for a value in an array
values produced by computations Postcondition: return value is the index of
some occurrence of the value, or -1 if the
Focuses on the data, while control-flow-
value does not occur in the array
based testing focuses on the control
CSE 757 - Software Testing 25 CSE 757 - Software Testing 26
Equivalence Partitioning Equivalence Classes
Consider input/output domains and Examples
partition them into equivalence classes Input x in range [a..b]: three classes “x<a”,
For different values from the same class, “a<=x<=b”, “b<x”
the software should behave equivalently boolean: classes “true” and “false”
Some classes may represent invalid input
Test values from each class
Example for input range 2..5: “less than 2”, Choosing test values
“between 2 and 5”, and “greater than 5” Choose a typical value in the middle of the
class(es) that represent valid input
Testing with values from different
classes is more likely to find errors than Choose values at the boundaries of classes
testing with values from the same class e.g. for [a..b], use a-1, a, a+1, b-1, b, b+1
CSE 757 - Software Testing 27 CSE 757 - Software Testing 28
Example Another Example
Spec says that the code accepts between Similarly for the output: exercise
4 and 24 inputs; each is a 3-digit integer boundary values
One partition: number of inputs Spec: the output is between 3 and 6
Classes “x<4”, “4<=x<=24”, “24<x” integers, each in the range 1000-2500
Chosen values: 3,4,5,14,23,24,25
Try to design inputs that produce
Another partition: integer values 3 outputs with value 1000
Classes: “x<100”, “100<=x<=999”, “999<x” 3 outputs with value 2500
Chosen values: 99,100,101,500,998,999,1000 6 outputs with value 1000
6 outputs with value 2500
CSE 757 - Software Testing 29 CSE 757 - Software Testing 30
5

7.
Basic Strategy for Unit Testing Test-First Programming
Create black-box tests Modern practices: the importance of unit
Based on the specification of the unit testing during development
Evaluate the tests using white-box Example: test-first programming
techniques (test adequacy criteria) Before you start writing any code, first
How well did the tests cover statements, write the tests for this code
branches, paths, DU-pairs, etc.? Write a little test code, write the
Many possible criteria; at the very least corresponding unit code, make sure it passes
need 100% branch coverage the tests, and then repeat
Create more tests for the inadequacies: E.g., used in Extreme Programming: any
e.g. to increase coverage of DU-pairs program feature without a corresponding
test simply doesn’t exist
CSE 757 - Software Testing 37 CSE 757 - Software Testing 38
Advantages of Test-First Programming System Testing
Developers don’t “skip” unit testing Goal: find whether the program does
Satisfying for the programmer: feeling of what the customer expects to see
accomplishment when the tests pass Black-box
Helps clarify interface and behavior From requirements analysis, there should
before programming be validation criteria
To write tests for something, first you need How are the developers and the customers
to understand it well going to agree that the software is OK?
Software evolution Many issues: functionality, performance,
After changing existing code, rerun the tests usability, portability, etc.
to gain confidence (regression testing)
CSE 757 - Software Testing 39 CSE 757 - Software Testing 40
System Testing (cont) System Testing (cont)
Initial system testing is done by the For multiple customers: two phases
software producer
Alpha testing: at the vendor’s site by a
Eventually need testing by the customers few customers
Customers are great testers
Beta testing: distributed to many end-
If the software is built for a single users
customer: series of acceptance tests Users run it in their own environment
Deploy in the customer environment and have Sometimes done by thousands of users
end-users run it
CSE 757 - Software Testing 41 CSE 757 - Software Testing 42
7

8.
Stress Testing Stress Testing (cont)
Form of system testing: behavior under Find how the system deals with overload
very heavy load
Reason 1: determine failure behavior
e.g. normally there are 1-2 interrupts per
If the load goes above the intended, how
second; what if there are 10 interrupts?
“gracefully” does the system fail?
e.g. what if data sets that are an order of
magnitude larger than normal? Reason 2: expose bugs that only occur
e.g. what if our server gets 10 times more under heavy loads
client requests than normally expected? Especially for OS, middleware, servers, etc.
e.g. memory leaks, incorrect resource
allocation and scheduling, race conditions
CSE 757 - Software Testing 43 CSE 757 - Software Testing 44
Regression Testing
Rerun old tests to see if anything was
“broken” by a change
Changes: bug fixes, module integration,
maintenance enhancements, etc. Testing of Object-Oriented Software
Need test automation tools
Load tests, execute them, check correctness
Everything has to be completely automatic
Could happen at any time: during initial
development or after deployment
CSE 757 - Software Testing 45 CSE 757 - Software Testing 46
Object-Oriented Software One Difference: Unit Testing
Widespread popularity in the last decade Traditional view of “unit”: a procedure
Initial hopes: it would be easier to test In OO: a method is similar to a procedure
OO software than procedural software But a method is part of a class, and is
Soon became clear that this is not true tightly coupled with other methods and
fields in the class
Some of the older testing techniques are
still useful The smallest testable unit is a class
It doesn’t make sense to test a method as a
New testing techniques are designed separate entity
specifically for OO software
Unit testing in OO = class testing
CSE 757 - Software Testing 47 CSE 757 - Software Testing 48
8

10.
State-based Testing Example: Stack
Natural representation with finite-state States
machines Initial: before creation
States correspond to certain values of the Empty: number of elements = 0
attributes Holding: number of elements >0, but less
Transitions correspond to methods than the max capacity
Full: number elements = max
FSM can be used as basis for testing
Final: after destruction
e.g. “drive” the class through all transitions,
and verify the response and the resulting Transitions: starting state, ending state,
state action that triggers the transition, and
possibly some response to the action
CSE 757 - Software Testing 55 CSE 757 - Software Testing 56
Examples of Transitions Finite State Machine for a Stack
Initial -> Empty: action = “create” Initial
e.g. “s = new Stack()” in Java
create add,delete
Empty -> Holding: action = “add”
add
Empty -> Full: action = “add” Empty Holding
delete
if max_capacity = 1 add delete
Empty -> Final: action = “destroy” add delete
destroy
e.g. destructor call in C++, garbage collection destroy
in Java Final destroy Full
Holding -> Empty: action = “delete”
CSE 757 - Software Testing 57 CSE 757 - Software Testing 58
FSM-based Testing Inheritance
Each valid transition should be tested People thought that inheritance will
Verify the resulting state using a state reduce the need for testing
inspector that has access to the internals of Claim 1: “If we have a well-tested superclass,
the class we can reuse its code (in subclasses, through
inheritance) without retesting inherited
Each invalid transition should be tested code”
to ensure that it is rejected and the
Claim 2: “A good-quality test suite used for a
state does not change superclass will also be good for a subclass”
e.g. Full -> Full is not allowed: we should call
add on a full stack Both claims are wrong
CSE 757 - Software Testing 59 CSE 757 - Software Testing 60
10

11.
Problems with Inheritance Testing of Inheritance
Incorrect initialization of superclass Principle: inherited methods should be
attributes by the subclass retested in the context of a subclass
Missing overriding methods Example 1: if we change some method m
Typical example: equals and clone is a superclass, we need to retest m
Direct access to superclass fields from inside all subclasses that inherit it
the subclass code
Example 2: if we add or change a
Can create subtle side effects that break
unsuspecting superclass methods subclass, we need to retest all methods
inherited from a superclass in the
A subclass violates an invariant from the context of the new/changed subclass
superclass, or creates an invalid state
CSE 757 - Software Testing 61 CSE 757 - Software Testing 62
Example Another Example
class A { class A {
int x; // invariant: x > 100 void m() { … m2(); … }
void m() { // correctness depends on void m2 { … } … }
// the invariant … } … } class B extends A {
class B extends A { void m2() { … } … }
void m2() { x = 1; … } … }
If inside B we override a method from A,
If m2 has a bug and breaks the invariant, this indirectly affects other methods
m is incorrect in the context of B, even inherited from A
though it is correct in A e.g. m now calls B.m2, not A.m2: so, we
cannot be sure that m is correct anymore
Therefore m should be retested on B objects and we need to retest it inside B
CSE 757 - Software Testing 63 CSE 757 - Software Testing 64
Testing of Inheritance (cont) Testing of Interacting Classes
Test cases for a method m defined in Until now we only talked about testing of
class X are not necessarily good for individual classes
retesting m in subclasses of X
Class testing is not sufficient
e.g. if m calls m2 in A, and then some
OO design: several classes collaborate to
subclass overrides m2, we have a completely
implement the desired functionality
new interaction
Still, it is essential to run all superclass A variety of methods for interaction
tests on a subclass testing
We will only consider testing based on UML
Goal: check behavioral conformance of the
interaction diagrams
subclass w.r.t. the superclass (LSP)
CSE 757 - Software Testing 65 CSE 757 - Software Testing 66
11

12.
UML Interaction Diagrams for Testing Collaboration Diagram
UML interaction diagrams: sequences of prepare()
messages among a set of objects :Order 1.2.1:needsReorder:=
There may be several diagrams showing needToReorder()
1*[all order lines]:
different variations of the interaction prepare()
*
Basic idea: run tests that cover all :OrderLine 1.1:hasStock:=check() :StockItem
diagrams, and all messages and conditions 1.2[hasStock]:remove()
inside each diagram 1.3[hasStock]:
If a diagram does not have conditions and create()
1.2.2[needsReorder]:create()
iteration, it contains only one path
:DeliveryItem :ReorderItem
CSE 757 - Software Testing 67 CSE 757 - Software Testing 68
Coverage Requirements
Run enough tests to cover all messages
and conditions
test with 0 loop iterations and >=1 iterations
test with hasStock=true and hasStock=false
test with needsReorder=true and
needsReorder=false
To cover each one: pick a particular path
in the diagram and “drive” the objects
through that path
CSE 757 - Software Testing 69
12