Tools

"... We present a technique that improves random test generation by incorporating feedback obtained from executing test inputs as they are created. Our technique builds inputs incrementally by randomly selecting a method call to apply and finding arguments from among previously-constructed inputs. As soo ..."

We present a technique that improves random test generation by incorporating feedback obtained from executing test inputs as they are created. Our technique builds inputs incrementally by randomly selecting a method call to apply and finding arguments from among previously-constructed inputs. As soon as an input is built, it is executed and checked against a set of contracts and filters. The result of the execution determines whether the input is redundant, illegal, contract-violating, or useful for generating more inputs. The technique outputs a test suite consisting of unit tests for the classes under test. Passing tests can be used to ensure that code contracts are preserved across program changes; failing tests (that violate one or more contract) point to potential errors that should be corrected. Our experimental results indicate that feedback-directed random test generation can outperform systematic and undirected random test generation, in terms of coverage and error detection. On four small but nontrivial data structures (used previously in the literature), our technique achieves higher or equal block and predicate coverage than model checking (with and without abstraction) and undirected random generation. On 14 large, widely-used libraries (comprising 780KLOC), feedback-directed random test generation finds many previously-unknown errors, not found by either model checking or undirected random generation. 1

...ome believe that in practice, random testing cannot be as effective as systematic testing because many interesting tests have very little chance of being created at random. Previous empirical studies =-=[9, 18, 28]-=- found that random test input generation achieves less code coverage than systematic generation techniques, including chaining [9], exhaustive genera1 tion [18], model checking, and symbolic execution...

"... We present a new purity and side effect analysis for Java programs. A method is pure if it does not mutate any location that exists in the program state right before the invocation of the method. Our analysis is built on top of a combined pointer and escape analysis, and is able to determine that m ..."

We present a new purity and side effect analysis for Java programs. A method is pure if it does not mutate any location that exists in the program state right before the invocation of the method. Our analysis is built on top of a combined pointer and escape analysis, and is able to determine that methods are pure even when the methods mutate the heap, provided they mutate only new objects. Our analysis provides useful information even for impure methods. In particular, it can recognize read-only parameters (a parameter is readonly if the method does not mutate any objects transitively reachable from the parameter) and safe parameters (a parameter is safe if it is read-only and the method does not create any new externally visible heap paths to objects transitively reachable from the parameter). The analysis can also generate regular expressions that characterize the externally visible heap locations that the method mutates. We have implemented our analysis and used it to analyze several applications. Our results show that our analysis effectively recognizes a variety of pure methods, including pure methods that allocate and mutate complex auxiliary data structures.

...id inter-procedural mappings between nodes of conflicting types. 5.1 Checking Purity of Data Structure Consistency Predicates We ran our analysis on several benchmarks borrowed from the Korat project =-=[3, 26]-=-. Korat is a tool that generates non-isomorphic test cases up to a finite bound. Korat’s input consists of 1) a type declaration of a data structure, 2) a finitization (e.g., at most 10 objects of typ...

"... TestEra is a framework for automated specification-based testing of Java programs. TestEra requires as input a Java method (in sourcecode or bytecode) , a formal specification of the pre- and post-conditions of that method, and a bound that limits the size of the test cases to be generated. Using th ..."

TestEra is a framework for automated specification-based testing of Java programs. TestEra requires as input a Java method (in sourcecode or bytecode) , a formal specification of the pre- and post-conditions of that method, and a bound that limits the size of the test cases to be generated. Using the method&apos;s pre-condition, TestEra automatically generates all nonisomorphic test inputs up to the given bound. It executes the method on each test input, and uses the method postcondition as an oracle to check the correctness of each output. Specifications are first-order logic formulae. As an enabling technology, TestEra uses the Alloy toolset, which provides an automatic SAT-based tool for analyzing first-order logic formulae. We have used TestEra to check several Java programs including an architecture for dynamic networks, the Alloy-alpha analyzer, a fault-tree analyzer, and methods from the Java Collection Framework.

"... Abstract—Bounded exhaustive testing (BET) is a verification technique in which software is automatically tested for all valid inputs up to specified size bounds. A particularly interesting case of BET arises in the context of systems that take structurally complex inputs. Early research suggests tha ..."

Abstract—Bounded exhaustive testing (BET) is a verification technique in which software is automatically tested for all valid inputs up to specified size bounds. A particularly interesting case of BET arises in the context of systems that take structurally complex inputs. Early research suggests that the BET approach can reveal faults in small systems with inputs of low structural complexity, but its potential utility for larger systems with more complex input structures remains unclear. We set out to test its utility on one such system. We used Alloy and TestEra to generate inputs to test the Galileo dynamic fault tree analysis tool, for which we already had both a formal specification of the input space and a test oracle. An initial attempt to generate inputs using a straightforward translation of our specification to Alloy did not work well. The generator failed to generate inputs to meaningful bounds. We developed an approach in which we factored the specification, used TestEra to generate abstract inputs based on one factor, and passed the results through a postprocessor that reincorporated information from the second factor. Using this technique, we were able to generate test inputs to meaningful bounds, and the inputs revealed nontrivial faults in the Galileo implementation, our specification, and our oracle. Our results suggest that BET, combined with specification abstraction and factoring techniques, could become a valuable addition to our verification toolkit and that further investigation is warranted. Index Terms—Formal methods, program verification, testing and debugging. 1

...[7], [34], [48]. Recent research has demonstrated the feasibility of employing bounded exhaustive testing (BET) to test small software modules, such as programs that manipulate simple data structures =-=[32]-=-. The underlying rationale for BET lies in Jackson’s small scope hypothesis [24]: that any given fault is likely to be revealed by testing some small input and that testing all inputs up to some small...

"... Modern software pervasively uses structurally complex data such as linked data structures. The standard approach to generating test suites for such software, manual generation of the inputs in the suite, is tedious and error-prone. This dissertation proposes a new approach for specifying properties ..."

Modern software pervasively uses structurally complex data such as linked data structures. The standard approach to generating test suites for such software, manual generation of the inputs in the suite, is tedious and error-prone. This dissertation proposes a new approach for specifying properties of structurally complex test inputs; presents a technique that automates generation of such inputs; describes the Korat tool that implements this technique for Java; and evaluates the effectiveness of Korat in testing a set of data-structure implementations. Our approach allows the developer to describe the properties of valid test inputs using a familiar implementation language such as Java. Specifically, the user provides an imperative predicate—a piece of code that returns a truth value—that returns true if the input satisfies the required property and false otherwise. Korat implements our technique for solving imperative predicates: given a predicate and a bound on the size of the predicate’s inputs, Korat automatically generates the bounded-exhaustive

...eturns true. More precisely, the solver produces only nonisomorphic structures that translate into a special kind of nonequivalent test inputs (Section 1.1.3). We have used bounded-exhaustive testing =-=[56,72,104]-=-, which tests the code for all inputs within the given small bound, to evaluate the effectiveness of our technique for generating structurally complex test inputs by solving imperative predicates. Mor...

"... Abstract. A test case consists of two parts: a test input to exercise the program under test and a test oracle to check the correctness of the test execution. A test oracle is often in the form of executable assertions such as in the JUnit testing framework. Manually generated test cases are valuabl ..."

Abstract. A test case consists of two parts: a test input to exercise the program under test and a test oracle to check the correctness of the test execution. A test oracle is often in the form of executable assertions such as in the JUnit testing framework. Manually generated test cases are valuable in exposing program faults in the current program version or regression faults in future program versions. However, manually generated test cases are often insufficient for assuring high software quality. We can then use an existing test-generation tool to generate new test inputs to augment the existing test suite. However, without specifications these automatically generated test inputs often do not have test oracles for exposing faults. In this paper, we have developed an automatic approach and its supporting tool, called Orstra, for augmenting an automatically generated unit-test suite with regression oracle checking. The augmented test suite has an improved capability of guarding against regression faults. In our new approach, Orstra first executes the test suite and collects the class under test’s object states exercised by the test suite. On collected object states, Orstra creates assertions for asserting behavior of the object states. On executed observer methods (public methods with non-void returns), Orstra also creates assertions for asserting their return values. Then later when the class is changed, the augmented test suite is executed to check whether assertion violations are reported. We have evaluated Orstra on augmenting automatically generated tests for eleven subjects taken from a variety of sources. The experimental results show that an automatically generated test suite’s fault-detection capability can be effectively improved after being augmented by Orstra. 1

...st suite before and after Orstra’s augmentation. Then our experiment requires faults for these eleven Java classes. These Java classes were not equipped with such faults; therefore, we used Ferastrau =-=[24]-=-, a Java mutation testing tool, to seed faults in these classes. Ferastrau modifies a single line of code in an original version in order to produce a faulty version. We configured Ferastrau to produc...

"... Forward symbolic execution is a program analysis technique that allows using symbolic inputs to explore program executions. The traditional applications of this technique have focused on programs that manipulate primitive data types, such as integer or boolean. Recent extensions have shown how to ha ..."

Forward symbolic execution is a program analysis technique that allows using symbolic inputs to explore program executions. The traditional applications of this technique have focused on programs that manipulate primitive data types, such as integer or boolean. Recent extensions have shown how to handle reference types at their representation level. The extensions have favorably been backed by advances in constraint solving technology, and together they have made symbolic execution applicable, at least in theory, to a large class of programs. In practice, however, the increased potential for applications has created significant issues with scalability of symbolic execution to programs of non-trivial size—the ensuing path conditions rapidly become unfeasibly complex. We present Dianju, a new technique that aims to address the scalability of symbolic execution. The fundamental idea in Dianju is to perform symbolic execution of commonly used library classes (such as strings, sets and maps) at the abstract level rather than the representation level. Dianju defines semantics of operations on away from the complexity that is normally inherent in library implementations, thus promising scalable analyses based on symbolic execution.

"... Managing access control policies in modern computer systems can be challenging and error-prone, especially when multiple access policies are combined to form new policies, possibly introducing unintended consequences. In this paper we present a framework for automated verification of access control ..."

Managing access control policies in modern computer systems can be challenging and error-prone, especially when multiple access policies are combined to form new policies, possibly introducing unintended consequences. In this paper we present a framework for automated verification of access control policies. We introduce a formal model for systematically specifying access to resources. We show that the access control policies in the XACML access control language can be translated to a simple form which partitions the input domain to four classes: permit, deny, error, and notapplicable. We present several ordering relations for access control policies which can be used to specify the properties of the policies and the relationships among them. We then show how to automatically check these ordering relations using an existing automated analysis tool. In particular, we translate XACML policies to the Alloy language and check their properties using the Alloy Analyzer. Our experimental results demonstrate that automated verification of XACML policies is feasible.

"... We present a new method purity analysis for Java programs. A method is pure if it does not mutate any location that exists in the program state right before method invocation. Our analysis is built on top of a combined pointer and escape analysis for Java programs and is capable of determining that ..."

We present a new method purity analysis for Java programs. A method is pure if it does not mutate any location that exists in the program state right before method invocation. Our analysis is built on top of a combined pointer and escape analysis for Java programs and is capable of determining that methods are pure even when the methods do heap mutation, provided that the mutation affects only objects created after the beginning of the method. Because our analysis extracts...