Testing requires:efficiency: finding bugs faster than normal useeffectiveness: finding bugs that users care about and that developers will fixthoroughness: leaving no stone unturnedWhere are the Bugs?To find them you could:Seek out the weak developers!Seek out the bad managers!Seek out the doomed projects!Create them yourself!Understanding where bugs are requires that we understand how and why software failsSoftware Fault ModelsFault models can be based on:Process maturityProgramming language constructsSoftware behaviorHow do we understand problematic behavior?Read bug reportsRecognize patterns of failureWhere Bugs Come FromEnvironmentThe software misinterprets or cannot handle its environmentWhat are all the environmental considerations that we must face?CapabilitiesThe software incorrectly performs one or more of its capabilitiesWhat are software’s general capabilities?

Environment Interactionis Complex!Let’s take a closer look at each of these interfacesThe Human InterfaceTwo main types of human interfaces:The Human InterfaceHow human inputs actually get transmitted to software is a point of interestThe demo shown is not only fun, it will demonstrate that there is more to input delivery than typing at a keyboardThe Human InterfaceTwo main types of human interfaces:

The Human InterfaceThe following example is an actual API developed by a leading software companyThe design of this API is likely not deliberate, instead it probably evolved over timeTesting this API will be a nightmareDebugging this API is probably impossibleThe Human InterfaceHere is a simple function prototype:The Kernel InterfaceThe operating system user is very complexThe Windows kernel exports over 1000 functionsEach function has at least two return valuesToo often, developers trust the kernel explicitlyThe Kernel InterfaceHow do we test the kernel interface?Run a bunch of background apps to:slow performancesimulate resource contentiontest low memory situationsBut…How do we repro these bugs?How can we debug the app?The Kernel InterfaceThe fundamental problem is that the kernel interface is reactiveWe need some good system tools to expose this interface to testers and give us the ability to proactively determine what return values our app seesThe demo will reveal just how much activity occurs at the kernel interfaceThe Software InterfaceExamples:Making SQL queries to an external databaseUsing a socket API to manage a network connectionUsing a math libraryUsing third party GUI controls

The Software InterfaceFact Run time libraries and reused components are SOFTWARE

Environment InteractionSoftware CapabilitiesNow that we understand software environments, lets take a close look at software capabilitiesUnderstanding CapabilitiesSoftware CapabilitiesAlthough there are only four basic capabilities:accepting inputproducing outputstoring dataperforming computationSoftware can combine these capabilities to perform very complicated tasksTesting Software CapabilitiesConfronting such complexity in a massive frontal testing assault is often unproductiveInstead, exploratory testers wage small (winnable) battles until the enemy submitsTesting is WarThe enemy: bugs in the softwareBugs prevent capabilitiesDrive capabilities and you find bugsMethod:Determine your enemy’s strengths and remove themWage small wars on input capabilities, output capabilities, data storage and computationTesting InputSoftware should only accept input that it can handle…

… But ensuring that this is the case is problematicTesting InputHow does software filter erroneous input?GUIsby preventing input data of incorrect typeby preventing input data that is too small or too largeby forcing the user down specific control flow paths

Testing InputHow does software filter erroneous input?Error checking code“if” statementsby ensuring that inputs received can actually be processedbut error checking code can also have errors! writing error code might introduce errorswriting error code also means diverting your attention from the main-line codeTesting InputHow does software filter erroneous input?Exception handlersfailing gracefully is extremely difficulterror routines must reset state and cleanup side-effects…a very difficult endeavorTesting OutputSoftware should generate only those outputs that are acceptable to its usersDisplayed data must fit in its display areaSoftware cannot pass incorrect data typesData must be correctly computed, software must never pass incorrect values to its usersTesting output requires domain expertiseWe must understand wrong answers and ensure that our software does not produce themTesting DataInputs and computation results are often stored internallySoftware will fail if it stores illegal dataStored data values must be acceptable individually and in combination with other dataTesting DataThe major difference between data and inputs is that data is persistentPersistence needs to be testeddata retrieval, data modification, data accesswe must test that data structures can be operated on without failureaccessed, retrieved, modified, overflowed, underflowed, …Testing ComputationSoftware can correctly filter inputs, validate outputs and store data… … and still failx=x+1 will fail if it is executed enough times to overflow the value xCorrect computation depends on operators, operands and resultTesting ComputationAnother aspect of computation is feature interactionFeatures can interact in ways that affect computationOne feature can get in the way of another feature’s computationSummary and ConclusionsSoftware TestingTesting consists of:Establishing an environment in which to execute the system under testThis means controlling:The user interfaceThe operating system kernelOther software systemsThe file system

Software TestingTesting consists of:Driving the system to exhibit its capabilitiesThis means exercising the software’s:Input domainOutput rangeInternally stored dataComputationNext Up: How to Break SoftwarePart 2How to Break SoftwareAct 1: The User InterfaceAn Overview of the MethodologySoftware possesses 4 basic capabilitiesAttack each capability by staging situations that commonly cause failureinput attacksoutput attacksdata attackscomputation attacksDetermine which attacks apply to your app and apply them, one-at-a-timeExploring the Input DomainBanging on the keyboard is largely a waste of time, a strategy for rookiesEach test should have a specific purposeA tester with clear goals is more likely to find a problem than a tester who is simply hacking awayTesters must learn to target problematic input scenariosExploring InputsExplore the application under testApply the inputs that a user would apply to get real work doneObserve the inputsWatch for attack opportunitiesPreview of Input Attacks1. Force all error messages to occur2. Force the software to establish default values3. Explore allowable character sets and data types4. Overflow input buffers5. Find inputs that interact with other inputs6. Repeatedly apply the same input/input sequenceDocumenting an AttackWhen to apply the attackWhat faults make the attack successfulHow to determine if the attack exposes failuresHow to conduct the attackExample and analysisFirst Attack: Force Error Messages to OccurWhen to apply this attack:Whenever an application must respond to erroneous inputTesters should consider the type of response:Input filterInput checkingException handlingForce Error Messages to OccurWhat faults make this attack successful:Programming error cases is difficultAdditional code must be written—where there is code, there is often bad codeWriting error code takes the programmers attention away from writing functional codeFailing gracefully is difficultWhat data needs to be saved?What state is the app in?Force Error Messages to OccurHow to determine if the software fails:This attacks findsMissing error casesNonsense error messagesUninformative error messagesThus, manifestation of the defect ranges from crash to fully functional code

Force Error Messages to OccurHow to conduct this attack:Look thru the (ahem) specification for message definitionConsider properties of inputs[type] entering invalid types often causes error messages[length] a few too many characters in an input string will often elicit an error message[boundary values] often represent special cases

Force Error Messages to OccurTarget: Word® 2000Feature: Insert IndexReproducibility: Works on all PC versions of Word® and all versions of Windows® 9x, NT, 2000Synopsis: Probably a duplicated block of code, very sloppy programmingForce Error Messages to OccurSecond Attack: Force Default ValuesWhen to apply this attack:All software uses variablesThe lifecycle of a variable isDeclarationInitializationUseVariables must be initialized before they are usedForce Default ValuesWhat faults make this attack successful:Compilers are often good at catching these mistakesBut the compilers must be used properlyImplicit declaration can often get a programmer into troubleWhen users skip input fields or leave them blank, defaults must be established in case those variables are used in computationForce Default ValuesHow to determine if the software fails:Best case (for ease of verification) is that use of an un-initialized variable causes a memory violationHarder to detect is that a random value gets assigned to the variableLook for garbage characters/”strange things”Incorrect resultsToo many values/too few values displayedIncorrectly typed data being displayedForce Default ValuesHow to conduct this attack:Determine the data that has defaultsLook for “options” or “configuration” buttonsConsult the source code’s declaration sectionForce the app to use its defaultsAccept any displayed data as defaultsEnter a null value, if a value is displayed then delete itChange settings from their default values to a valid value and then back againForce Default ValuesTarget: Word® 2000Feature: Insert Table of ContentsReproducibility: Works on all PC versions of Word® and all versions of Windows® 9x, NT, 2000Synopsis: Frankly, I would have difficulty coding this behavior if I wanted to!Force Default ValuesThird Attack: Explore Character Sets/Data TypesWhen to apply this attack:Target: variable inputSpecial cases based on:Operating system reserved wordsProgramming language reserved wordsCharacter set boundaries (ASCII, UNICODE, etc.)spaces, quotation marks, delimiters, etc.Explore Character Sets/Data TypesWhat faults make this attack successful:Special cases require special handlingEither:Developers fail to recognize a special caseDevelopers put too much trust in interface controlsDevelopers fail to handle errors properly (we’ve already discussed that error code is hard to get right)

Explore Character Sets/Data TypesHow to determine if the software fails:Unhandled exceptions cause a system to crashGeneralized error handlers often present nonsensical or uninformative messagesWatch out for loss-of-state often caused by exception handlersSince we are dealing with character data, watch for rendering problemsWatch for “Easter eggs”Explore Character Sets/Data TypesHow to conduct this attack:Spend some time researching:Operating system, programming language and character set keywords and rangesStudy documentation of each of the aboveBeware outdated keywordsExplore Character Sets/Data TypesTarget: Microsoft® Internet Explorer® v5.5Feature: File OpenReproducibility: Works only on early or recent versions of IESynopsis: AUX is an old DOS device, device names were not considered when the file open program was createdExplore Character Sets/Data TypesFourth Attack: Overflow Input BuffersWhen to apply this attack:The idea is simple: enter long strings into input fieldsDon’t neglect APIs/exposed internal objectsThis is the hacker’s choice because many buffer overflows create exploitable failure scenariosFourth Attack: Overflow Input BuffersWhen to apply this attack:This is an important bug because:copy/paste into inputs fields is a fairly common practiceBuffer overruns result in crashes, risking data loss and costly reworkbe sure to check with developers to find out tolerance for fixing long string bugs…thousands characters can get ridiculous

Overflow Input BuffersWhat faults make this attack successful:Developers simply fail to constrain the amount of text the software will accept in an input stingWhen the text is read input memory, fixed-sized buffers are overflowed

Overflow Input BuffersHow to determine if the software fails:This bug almost always causes the software to crashOther possibilities are extreme application instability (since memory is often overwritten by the long string)Overflow Input BuffersHow to conduct this attack:Identify where strings are read as inputStart small and then grow the string to its maximum lengthThis attack is very repetitiveIt’s helpful to count 12345678901234567890Overflow Input BuffersTarget: Word® 2000Feature: Find/ReplaceReproducibility: Works on all PC versions of Word® and all versions of Windows® 9x, NT, 2000Synopsis: Find field is string-length constrained but the Replace field is not

Overflow Input BuffersFifth Attack: Find Interacting InputsUp to now, we have dealt with inputs one-at-a-timeFind a spot where input is accepted and poke it until something breaksThis next attack deals with combinations of inputsMultiple inputs on a single input dialogAn API call with more than one inputConstraining a single input is hard enough for developers...Fifth Attack: Find Interacting InputsWhen to apply this attack:Some inputs affect other inputsThey might represent different properties of the same piece of dataThey might be used in a single internal computationThus, individually correct inputs might be problematic when combinedThese relationships should be watched for while executing the previous attacksFind Interacting InputsWhat faults make this attack successful:Obviously, input relationships are hard to determine for both testers and developersThe logic involved in handling a single erroneous input is hard enough……multiple error cases often require complex nested IF statementsCode changes make this situation worseFind Interacting InputsHow to determine if the software fails:Since inputs are often stored internally, slipping a bad input in means corruption of internal dataOnce you suspect you tricked the app into accepting bad input, force that input to be used as much as possibleCarefully verify it every time it is displayed or usedFind Interacting InputsHow to conduct this attack:Explore the app, identify possible relationships between inputsAre they properties of a single data structure?Are they used together in a single computation?Select boundary and extreme combinations:Large/small, small/large, large/large, small/smallFind Interacting InputsTarget: Word® 2000Feature: Insert a TableReproducibility: Works on all PC versions of Word® and all versions of Windows® 9x, NT, 2000Synopsis: These two parameters can be small-large, large-small, small-small but not large-largeFind Interacting InputsSixth Attack: Repeat Inputs Numerous TimesWhen to apply this attack:This attack is applicable whenever the app accepts input inside a loop:

Accept an inputProcess itRepeat

Repeat Inputs Numerous TimesWhat faults make this attack successful:Repetition has the effect of gobbling up resourcesMany applications are unaware of available resources and assume unlimited memory and storageLow resources can cause undesirable side-effectsRepeat Inputs Numerous TimesHow to determine if the software fails:It is difficult to predict how memory stress will manifestWatch for:Screen refresh problemsSlow performanceConsider:Using a memory leak detectorRepeat Inputs Numerous TimesHow to conduct this attack:Pick an input or a sequence of inputsApply them over and over to test for undesirable side-effectsOr:Pick an object and apply the same input to it over and overPick multiple objects and apply the same series of inputs to each objectRepeat Inputs Numerous TimesTarget: Word®Feature: Equation EditorReproducibility: Works on all PC versions of Word® and all versions of Windows® NT, 2000Synopsis: The number of parentheses is constrained at 10, but that number should, perhaps, be loweredRepeat Inputs Numerous TimesReview of Input Attacks1. Force all error messages to occur2. Force the software to establish default values3. Explore allowable character sets and data types4. Overflow input buffers5. Find inputs that interact with other inputs6. Repeatedly apply the same input/input sequenceExploring OutputsSome bugs are too difficult to find by concentrating on inputs aloneWhich inputs will generate incorrect results?Why not concentrate on what incorrect results could occur and then find the inputs to force them?Preview of Output Attacks7. Force different outputs to be generated for each input8. Force invalid outputs to be generated9. Force output properties to change10. Force the screen to be refreshedSeventh Attack: Vary Outputs for Each InputWhen to apply this attack:Inputs are not independent of each otherApplying some inputs before (or after) other inputs can affect behaviorA good way to think about the most important cases is to ensure that you see every output that an input can causeVary Outputs for Each InputWhat faults make this attack successful:Input ? Output is simple to codeInputs that cause different outputs require complex logic to be codedComplexity leads to bugs

Vary Outputs for Each InputHow to determine if the software fails:The main bug is that there are behaviors that the developer miscoded or forgot to codeThese are usually severity 1 problems that get found and fixed before releaseVary Outputs for Each InputHow to conduct this attack:Testers must identify which inputs can cause multiple behaviors and understand the context in which these inputs can be appliedVary Outputs for Each InputTo illustrate, consider a telephone switchHow many outputs can the input “pick up the receiver” cause?The switch could generate a dial tone … … when the phone is idleThe switch could connect two callers… … when the phone is ringingVary Outputs for Each InputEighth Attack: Force Invalid OutputsWhen to apply this attack:Domain expertise is essential to effectively carry out this next attackIt’s hard to test a flight simulator without knowing how to fly an airplaneTesters must know the product well enough to enumerate wrong answers……and then figure out how to drive the application to produce themForce Invalid OutputsWhat faults make this attack successful:Just as testers can misunderstand the problem domain…so can developersWhen this happens, they write bugsIn most cases, the problem is over-looked special casesForce Invalid OutputsHow to determine if the software fails:This is a hard one…the software rarely fails in a spectacular (or even noticeable) mannerTesters should ignore type and format and concentrate instead on the value being displayedForce Invalid OutputsHow to conduct this attack:Testers need to focus on known bad resultsEnumerate wrong answers and then create the situation that forces themLearn as much about the problem domain as you canForce Invalid OutputsTarget: Windows NT System ClockFeature: CalendarReproducibility: Works on Windows® NT, Service Pack 4 and priorSynopsis: The calendar works fine as long as the cursor isn’t fixed on day 29 of FebruaryForce Invalid OutputsForce Invalid OutputsNinth Attack: Change Output PropertiesWhen to apply this attack:This attack requires persistent outputs be present in the application (which often precludes its use on APIs)An output is generated and then we change some property of the output

Change Output PropertiesWhat faults make this attack successful:Generating the output once tests that the software works with initial data settingsThese are settings that the developer establishedThese are settings that the developer has anticipatedChanging the output ensures that the software will work user-defined settingThese settings might not be anticipatedChange Output PropertiesHow to determine if the software fails:Since outputs are necessarily visually-intensive, this often requires manual verificationBut since testers have had to determine output properties in advance, it is easy to determine what to look forChange Output PropertiesHow to conduct this attack:First determine the property of interestSize, value, type, color, shape, direction, …Cause it to be displayed, then change itSize (bigger to smaller, smaller to bigger,...)Value(s) (high to low, positive to negative,...)Type (char to int, int to float,...)…Change Output PropertiesTarget: PowerPoint® 2000Features: Insert WordArtReproducibility: Works on all PC versions of PowerPoint® and Word® and all versions of Windows®Synopsis: First time through, two things happen; next time through, something is forgottenChange Output PropertiesTenth Attack: Force the Screen to RefreshWhen to apply this attack:This attack is applicable only to GUI software with editable display areasThe attack is most useful at the boundaries of screen objectsObjects created by the userPartitions of the display area (frames, etc.) set by the softwareForce the Screen to RefreshWhat faults make this attack successful:Refreshing the contents of a window, after those contents have changed, is problematicRefresh too often:Performance degradesThe screen flickers to annoy the userRefresh too seldomThe screen becomes messyHitting the sweet spot is challenging

Force the Screen to RefreshHow to determine if the software fails:Sigh…you guessed it…labor-intensive visual verification is the only resortForce the Screen to RefreshHow to conduct this attack:Add things to the screenMove them around (varying the distance you move them)Delete themEdit themDo all these things with a mix of features and at or around object boundariesForce the Screen to RefreshTarget: Word® 2000Features: Entering text at the page boundaryReproducibility: Works on Word® 2000 on any OS. The problem is much worse on prior versions of WordSynopsis: Removal of a similar bug in Word 97 caused this oneForce the Screen to RefreshReview of Output AttacksExploring Stored DataWhile executing the input and output attacks, keep notes on persistent (stored) dataData you see over and over is storedData you see over multiple executions is persistentThink about where this data comes from and how it can get corruptedExploring Stored DataIf you have the source code, data is easy to findIf you don’t have the source, you must be able to look through the interface and identify dataPut yourself in the shoes of the developerHow would you program it?Preview of Stored Data Attacks11. Apply inputs using a variety of initial conditions12. Force a data structure to store too many/too few values13. Investigate alternate ways to modify internal data constraintsEleventh Attack: Vary Initial ConditionsWhen to apply this attack:Inputs are often applicable in a variety of circumstances (states of the software)This attack investigates the application of inputs from a variety of initial statesTry to pick states that might cause errant behaviorPick states that are as different from each other as possiblePick states that differ by only one data elementVary Initial ConditionsWhat faults make this attack successful:Checking that inputs are valid is trickyChecking that inputs—combined with internal state – are valid is even harderThe result is that some inputs only work on specific initial conditionsVary Initial ConditionsHow to determine if the software fails:Since some cases of this bug mean missing code (failure to consider a specific case)Watch for crashesWatch for nonsense error messagesVary Initial ConditionsHow to conduct this attack:Enumerating states is a time consuming endeavor:Does an input cause different behavior in different circumstances?Is an input applicable sometimes but not other times?For more information read about model-based testingvisit: www.model-based-testing.orgVary Initial ConditionsTarget: Word® 2000Features: Draw Group/UngroupReproducibility: Works on all PC versions of Word® and all versions of Windows® 9x, NT, 2000Synopsis: Certain configurations of grouped objects do not ungroup cleanlyVary Initial ConditionsTwelfth Attack: Over/Underflow Data StructuresWhen to apply this attack:Fixed size structures can be overflowed by forcing the software to store too many valuesTesters must be able to detect size constraints and then find ways to violate themOver/Underflow Data StructuresWhat faults make this attack successful:Developers are often good at checking content and failing to consider sizeEvery program that adds or removes elements from a structure must have code to check that size constraints are not broken as a result of the add/remove operationIt only takes one program to forget…Over/Underflow Data StructuresHow to determine if the software fails:Reading or writing beyond the bounds of a data structures almost always causes the software to crashBut be alert for the usual signs of data corruption (performance, anomalous output/computation, etc.)Over/Underflow Data StructuresHow to conduct this attack:A structure can be underflowed by removing more values than the structure containsStrategy:add x number of values to a structuredelete x+1 values from the structureTry this on internal structures and also list boxes that store persistent dataOver/Underflow Data StructuresTarget: Word® 2000Features: TablesReproducibility: Works on all PC versions of Word® and all versions of Windows® 9xSynopsis: failure to constrain the upper bound on the number of rowsOver/Underflow Data StructuresThirteenth Attack: Find Back Doors to Stored DataWhen to apply this attack:Whenever data is supposed to be constrained in any way:A month must fall between 1 and 12A year must fall between 1980 and 2095The limit on undo operations is 20…Find Back Doors to Stored DataWhat faults make this attack successful:Plain and simple: a lack of good software engineering practiceInformation hiding (object-orientation) cures this problem:All data is private to a set of access routinesThese access routines enforce data constraintsA program must use the access routines to access or modify dataFind Back Doors to Stored DataHow to determine if the software fails:Broken data constraints are serious, often resulting in system instability or crashAlso, look for:System sluggishnessIncorrect error messagesIncorrectly computed resultsFind Back Doors to Stored DataHow to conduct this attack:Identify data structuresExplore various ways to modify the contents or properties of the data structure

Find Back Doors to Stored DataTarget: PowerPoint® 2000Features: Insert TableReproducibility: Works on all PC versions of PowerPoint® and all versions of Windows® 9x, NT, 2000Synopsis: Table size is constrained at creation but can be expanded on editFind Back Doors to Stored DataReview of Stored Data AttacksExploring ComputationWhile executing the input and output attacks:make a feature listkeep notes on what computation is happening inside the applicationThink about ways to “get in the way” of the computation, using inputs, outputs, data, or even other featuresPreview of Computation Attacks14. Experiment with invalid operand and operator combinations15. Exploit recursion16. Force computation results to be too large or too small17. Find features that share data or interact poorlyFourteenth Attack: Experiment w/ Operator/OperandsWhen to apply this attack:Some operands cannot be used with certain operatorsDivide by zeroAdding a character to a real numberSome operators conflict with each otherOperators with an inverse relationshipEx: [SQRT(x)]2Experiment w/ Operator/OperandsWhat faults make this attack successful:Developers have to write error code to weed out invalid operator/operand combinationsConsider divide-by-zero

if (x != 0) y = z / x;else cout(“cannot divide by zero”);Experiment w/ Operator/OperandsHow to determine if the software fails:Become a problem domain expert to learn about the domain and range of the functions computed by your softwareApply invalid combinations and watch for:Crashes (for unhandled exceptions)Incorrect resultsNull or nonsensical values appearing as outputExperiment w/ Operator/OperandsHow to conduct this attack:Must be able to identify computation (operators) and the data objects (operands) on which the computation operatesComputation isn’t just assignmentsGraphical renderingScreen layout, WYSIWYGExperiment w/ Operator/OperandsTarget: Windows® CalculatorFeatures: Square Root/ExponentReproducibility: Works on all versions of Windows® 9x, NT, 2000Synopsis: Round-off errorExperiment w/ Operator/OperandsFifteenth Attack: Exploit RecursionWhen to apply this attack:Recursion occurs when a function calls itself

If this occurs too many times, stack overflow will occurExploit RecursionWhat faults make this attack successful:Developers fail to write code that ensures loop or recursive call terminationSince recursion only requires one line of code, constraining it often seems unnecessary to developers

Exploit RecursionHow to determine if the software fails:Infinite recursion causes a heap overflow in the computer’s memoryThis will almost always result in the application crashing or hangingExploit RecursionHow to conduct this attack:Recursion can be thought about in terms of black box objects too:Can a document reference itself?Can a hyperlink point to itself?Can an email message contain itself?Can a program that spawns processes call itself?Determine these objects and create the self-referencing situationExploit RecursionThis demo will shows how such a program behaves on Windows 2000...

…However, we will wait until the end of the session because the computer will not be usable after this program executesExploit RecursionSixteenth Attack: Force Results to be too Large/SmallWhen to apply this attack:Review:Inputs need to be constrainedan app should never permit invalid input to enter the systemData needs to be constrainedan app should never store invalid dataBut even if all inputs and data are correct, computation can still failForce Results to be too Large/SmallWhat faults make this attack successful:

x=x+1;

This simple line of code will fail if it is executed enough timesDevelopers often focus on operators and operands and ignore the resultForce Results to be too Large/SmallHow to determine if the software fails:Since the result of this attack is underflow or overflow, the software most often crashesForce Results to be too Large/SmallHow to conduct this attack:Usually, you have to force a computation to occur over and over and overIf you have control over the data used in the computation, rig it to begin at or around boundary valuesForce Results to be too Large/SmallGiven the program:#define count 2main() { int sum, value[count]; sum = 0; for (i = 0; i < count; ++i) { sum = sum + value[i]; }}Consider the values:value[0] = 32700, value[1] = 70count = 33000, value[0..32999] = 1Force Results to be too Large/SmallSeventeenth Attack: Feature InteractionWhen to apply this attack:This is the attack that distinguishes novice testers from the prosGiven two or more features that work fine individually, can you make them break when they work togetheron the same dataat the same timegetting in the way of each otherFeature InteractionWhat faults make this attack successful:Features often work well in isolation because they are developed in isolationBut two features may share data but require that different constraints on that data be enforcedFailure to enforce such constraints causes this class of failureFeature InteractionHow to determine if the software fails:Here is yet another example of painstaking manual verificationKeep a sharp eye out for:Improperly formatted outputIncorrect computationCorrupted dataFailures often manifest long after the fault has been tripped overFeature InteractionHow to conduct this attack:Pick two or more featuresThat might interact with the same user inputThat might contribute to computing a single outputThat might share the same dataTry to make them “get in each other’s way”

Feature InteractionTarget: Word® 2000Features: Footnotes/Dual ColumnsReproducibility: Works on all PC versions of Word® and all versions of Windows® 9x, NT, 2000Synopsis: The footnote adopts the properties of the reference point, which can be incompatible with the page layoutReview of Computation AttacksThe User InterfaceHow to Break SoftwareAct 2: The Kernel InterfaceThe Kernel InterfaceAlthough the kernel is responsible for numerous application services, it’s most central task is memory managementHow can memory cause applications to fail?The system can run out of memoryApps can contend for resourcesThe operating system can failTesting the Kernel InterfaceTesting OS and memory failures is problematic:Running dozens of background apps is time-consuming and requires significant monitoring and cleanupDebugging root cause is guess work…which calls are actually failing?Kernel ImpostersHolodeck uses a kernel imposter that:logs all app-to-kernel activityallows kernel-to-app return values to be controlled by the testerHow to Effectively Fault Inject the Kernel Interface...…Is an open questionCertainly:It is environment specifichandhelds, embedded systems, desktop apps all have different thresholds of acceptable behaviorIt is application specificsome apps must be fault tolerant or provide a high degree of safety or securitysome apps can fail and nobody cares, including the developersFault Injection StrategyThere are two approaches:Monitor your application in a stressed environmentUse Holodeck to record the traffic across the kernel interfaceSimulate a similar pattern of failed calls for testing successive builds of the same productFault Injection StrategyThere are two approaches:Systematically fail each call and document the application’s behaviorGet with developers to analyze root cause and decide remedies (if any)Thoroughly test each successive build for problematic callsHow to Break SoftwareAct 3: The File System InterfaceThe File System InterfaceNearly all applications use the file systemHow can the file system cause applications to fail?Categories of potential problems:Media problemsFile problemsMedia-based AttacksFill the file system to its capacityForce the media to be busy or unavailableDamage the mediaFile-based AttacksAssign an invalid file nameToo longInvalid character set/formatVary access permissionsRead only, user privilegesVary/corrupt file contents

How to Break SoftwareAct 4: The Software InterfaceThe Software InterfaceMost complex applications are built from components, many of them reusedHow can reused software cause our application to fail?Reused software is software…we should expect it to fail!It can pass unexpected return valuesIt can pass unexpected or corrupt dataFault Injecting the Software InterfaceSame two approaches for the kernel interface apply also to software:Monitor the calls…inject realistic faults…simulate by replaying the interesting scenariosSystematically fail each callSummarySoftware testing requires understanding of and control over four interfaces:UIKernelFile SystemSoftwareSummaryEach interface presents number of challenging problemsTesting any of the interfaces is formidableTesting them thoroughly is exquisitely difficultSome interfaces are well-understood, some we are just beginning to studyConclusionsTesting each interface:The user interfacestudy the attacksexecute the attacksmeet often, share best practices

ConclusionsTesting each interface:The kernel interfaceacquire the toolsacquire the knowledgestudy field bug reports and make sure you understand real stressed environments

Testing each interface:The software interfaceunderstand the other apps that your software depends uponunderstand the data that your app gets from these resourcesdetermine how this communication can faildetermine if failure of the resource will cause failure of your appRememberKnow your app’s environmentUnderstand what might go wrong and test that it doesn’tKnow your app’s capabilitiesUnderstand how it can screw things up, test that it doesn’tPractice the attacks…always have a goalBrain on, eyes open, Test!