A simple example is used to show how to use Aspect Oriented Programming to provide fault injection.

Intro

A few months ago I was looking at Java code and thinking there must be a better way to see if this will really work, if there are any issues in how the program would react to exceptional conditions. I could write some unit tests or force some problems to see if the exception handling is correct and the log file output is really useful. But is there another way?

I got a solution after a while of thinking: Insert faults. Now how to do that in an easy and maintainable way? Not a new idea of course. After doing a web search I found some interesting references.

Funny, today at work I was extending an application and one of the support objects was failing. Turned out to be an index out of bounds problem. The method had adequate exception handling but, of course, an index out of bounds scenario was not tested. In this case the problem would not occur in production (hmmm), but it definitely occurred during development, and was not handled correctly.

Exception Handling

There are many references of correct exception handling strategies and best practices. However, when creating applications, the developer must still make critical creative decisions about how to handle an exception. That decision may not be a wise one. And even if it is, may later prove to have been very unwise since the system requirements and implementation may have changed due to maintenance or requirement evolution.

Testing Approaches

Using various testing methodologies a subsystem can be exhaustively tested for exception handling. Yet, most testing methods require that the subsystems be isolated in some manner. This, though very valuable, can give fatal false confidence that the system will behave in a predictable manner in responding to exceptional conditions.

The prime example of this is within the Unit Testing methodologies. Since Unit Tests are tests of isolated components, they do not test actual exception handling in the live system. A perfect component can still contribute to system failure when it is used incorrectly or uses other components incorrectly. For example, a unit test can show that class X’s methods correctly handle and when necessary throw the perfect exception. That means nothing if components that interact with X don’t correctly use those exceptions: they swallow, mask, or mistakenly catch them when there is nothing they can do.

Thus, one needs to use Integration Testing to test assemblages of components. But, we are still back with the same shortcoming as with Unit Testing. Thus we would next need various forms of system testing. And, that system testing must be a White Box test. A Black Box test wherein an interface is exercised with various inputs (such as in web test systems or by QA personnel), will not necessarily invoke the full internal API/SPI between components that could be part of a complicated call chain. Why? Because, the interface, such as a browser client, will have (we hope) the validation, security, and logging layers implemented so that by the time a system test workflow enters the deep system components within the client or middleware, there is no way to influence the target component state into programmatic intentional failure mode.

If there is no way to exercise a run time trajectory, then why bother? The reason is that exceptional conditions are exceptional. Resources can be exhausted, systems not available, and so forth. The rule of Fail Fast may not be enough if collaborating components are not exceptionally responsive.

Fault Injection

To see if a system is responding to exceptional conditions, one must wait for those conditions or create them. Analogously to Fuzz Testing we can dynamically and randomly insert faults. In Fuzz Testing, inputs are manipulated to a system under test. In Fault Injection we have to manipulate inputs inside the real system, within the components themselves. And as in Fuzz Testing, we can employ various strategies to do so, such as randomized faults, etc. Of course, this system is not the deployed production system, it is the real system in a test environment, an internal full deployment.

Some approaches are: source modification, source macros, annotations, dependency injection (IOC), and Aspect Oriented Programming (AOP). Of course, there are many other techniques as found in open/closed projects, commercial products, and the research community.

Fault Injection using AOP

Since using the real system is required, a viable approach is to use the Instrumentation already available in the Java system. We could dynamically insert code that forces exceptions. AOP as implemented by the AspectJ language can already do this.

The major advantage of using AOP is that the source is not changed in any way. AspectJ is the state of the art in the Java ecosystem for using AOP.

Other approaches

Monitoring

BTrace (a Java oriented approach comparable to DTrace) is very interesting. However, since, like DTrace, it is meant to be safe, defect injection may not be doable. But, see this post on an unsafe mode for BTrace.

Source modification

One way of injecting failure is just to edit the source code and add strategic code that will cause the instigating failure. An advanced form of this is Mutation Testing. Code changes could be: arguments with wrong values, nulls, deliberately thrown exceptions, etc. For example, one can simply create a method that throws a runtime exception:

Then insert invocations to this method at strategic points in the code, deploy to a test environment, and see if the expected results are obtained, such as logging output that can identify the cause, or that no side effects are recorded, such as incorrect data storage.

This is a low tech and error prone approach. Most importantly, this is dangerous. In the mad development rush to meet deadlines and also eat lunch, this code could make it into production! Even with that horrific method name, it will wind up in production! One would have to create deployment filters that stop the process if any “fault” inducing code is included. Another reason why this is not a good approach is that it doesn’t scale very well. Forcing exceptions is just one type of verification. Another is setting values outside of expected ranges or states. Thus, one would need many different kinds of source code insertions.

Of course, Unit and Behavior based testing tools sets can supply these required verifications if included into the system as Built-In Self-Tests (BIST).

Built-In Self-Test

In the hardware realm, BIST as found in, for example, IEEE 1149.1 JTAG, has been very successful. On the software side, there is ongoing research on how to implement BIST-like capability. This would make Brad Cox’s concept of the “Software IC” even more powerful.

Macros

A somewhat viable approach to source code fault insertion is instead of inserting the faulting code, insert “include” macros in the original source code that indicates what fault insertion should be done at a location. A fault injection preprocessor can scan and insert the fault framework’s invocations to accomplish the requirement. The source code build process can then simply enable or disable the use of the preprocessor. This would however still require source code modification and an extra maintenance nightmare. When the code changes the macros may also require change.

Annotations

Instead of macros, we could also use Annotations. Annotations could explicitly state the runtime exceptional behavior “service level agreements” at the component level. These SLA could then be systematically manipulated to test if they really hold at the system level.

Dependency Injection

One can also inject exceptional behavior by dependency injection, using programmatic, declarative, or annotations, faulted components could be inserted (or created via Aspect Oriented Programming) into the target system.

Running at the command line

Using AspectJ is much easier in a supporting IDE like Eclipse. Below is the “mess”, unless you love the CLI, of compiling and running in a command shell. Could be made clean by creating aliases, scripts, etc.

Recently I brought up the topic of Aspect Oriented Programming (AOP) at work. I argued that it does have uses beyond mere tracing and psuedo-logging. In fact, it’s all over the place.

Just to get familiar with it again I looked at how it could be used on a test scenario I had. A class under test directly uses a Singleton collaborating class to invoke a service call, in other words non-dependency injection. This is similar to a class using “new” to create objects it depends on.

A requirement of unit testing is the isolation of the class from its collaborators (to a pragmatic degree). Since the Singleton class has a private constructor it can’t be subclassed or proxied (even with cglib via Mock frameworks).

I created a simplified version of the scenario to show how “easy” it is to use AOP. First the class under test class in listing one:

Now, the scenario goal is to change the speak() method to instead output “Goodbye!”. The aspect in listing three accomplishes this. It defines two pointcuts, cut() that specifies the class under test, and service() that identifies the target speak() method call in the collaborator object. The around() advice allows us to substitute our own replacement ‘method’. In the ‘around’ advice, using proceed() would let the original method execute. Note the terminology used here is not Aspectively correct.

Of course, our aspect in real life would not “do” the actual required test actions, but instead delegate to a test class or stub to do so. This is especially important if the collaborator object has many methods that must be advised. At first I was trying to do this directly, swapping out the use of the Singleton Collaborator class using static crosscutting, to another dynamically generated stub or mock class. Though with AspectJ you can do some powerful stuff, the private constructor puts a kink in all that.

Left to the reader is actually making this work within an actual Unit Testing framework like JUnit.

Updates

30 Oct 2010: Another approach which in many cases may be better is using the JMockit framework. It does work with real Singletons and other cases where AspectJ would be the only recourse.

15 Dec 2010: If you can change the source code, a very simple test pattern is to extract the troublesome code lines into a method. Now that method can more easily be mocked.

Interestingly the author correctly mentions that some languages now have rudimentary support for handling of basic crosscutting concerns. And, since the analyzed projects did not really need advanced AOP techniques,

“numbers indicate that languages that provide only basic crosscutting mechanisms are appropriate to implement a large extent of the analyzed programs.”

Note that the study does not apply to container based AOP as used in frameworks such as JBossAOP and SpringAOP. Several Dynamic Languages use a Metaobject Protocol.

I used AspectJ for some utility and diagnostic help. The language is fine and the concepts are approachable. It’s the configuration, tools, and environment hassles that put a damper on things. That’s why frameworks and tool support like that found in Eclipse are critical.