Meta

I’m currently involved with a WPF project as a tester. My charge is to automate the System-Level testing of this project for my client. Our belife is that automated testing allows us to find “popup bugs” before they become a problem. This has worked well, in combination with the team doing a degree of dog-fooding. They catch the esoteric things, I find the simple things no one looks for. Also, I’m currently working with a lot of them as we build manual test scripts for our verification phase. This is a lot more focused act that, once automated as we move forward with version two, will allow us to have a better understanding of the “shipability” of our software. We’re regulated and we want to automate most of the acceptance testing, at least in theory.

I’ve been using TestComplete for this. T.C. is good stuff, but there are some things about it I don’t like. It’s meant for recording and playback of tests. But, to automate in a regulated situation, I don’t want to see javascript (T.C.’s CSharp script is a horrible trap, don’t go there) like the following

What the hell is my intent here? Also, I’m crawling the visual tree, which may be brittle. The renaming of internal variables, like “StackPannel” to something more revealing like “VisibleFlarns,” I’d have to go edit this test. And a buch of other tests that traverse that point in the tree. We kind of saw this coming and have rewritten our tests to a degree.

This is slightly more clear. The main benefit is that we’re hiding the traversal of the tree to the visible flarns. That means we only have one place to hunt down our changes to the tree. But it doesn’t take full advantage of our environment, JavaScript. Foreign objects, like the process, windows, and buttons can’t be extended with helper methods. So, I’ve started the next round of testing with a bunch of decorators and strict reuse.

var process = new MyProcess();
process.GetVisibleFlarn(2).Cancel();

That’s starting to get to good. The last piece is that regulatory oversight loves positive passing of tests, not just error logs. T.C. doesn’t have that. T.C. doesn’t even really have asserts. You can only error, or log. In fact, I’ve yet to find a test framework that does that out of the box. Even those that assert, don’t log positive asserts. Why is this important? I want to be able to say Req-123 was tested 7 times, and passed each one. That’s what the oversight wants. They fear, “Well, it’s didn’t show up in the error logs, so I’m assuming it worked.” So, I wrote some asserts that take explicit requirements and “Why” I’m asking about that.

Now, on to prong two, of my assault.

I recently picked up “Scripted GUI Testing with Ruby” by Ian Dees as I visited my local book store. I like the book. It covers using Win32API to grab window handles and throw around events. Oh, it’s also got some crap about Jruby and an apple-talk bridge if you want to go that route. Basically, it covers testing on all the platforms that aren’t X11 or .NET. That doesn’t detract from the book’s handling of GUI testing strategy, but I want to fill in that gap.

The CLR has a pretty good testing story, for WPF at least, in the White testing framework from ThoughtWorks. It uses Automation Ids to identitfy and find all of your controls. So, the pieces I’m looking at are, “What are Automation Ids,” and “What are their best practices?” Couple that with a running example of using IronRuby to get my hands on the application and bang it around. The big deal is our current project doesn’t use Automation Ids (accessibiltiy isn’t a huge driving need), so I need to show some benefit and low risk.

What am I going to have to do, for this experiement? I’m, currently, only interested in rSpec and don’t need cucumber. I’ll start by using White directly, but I’m guessing I’ll be wrapping it some. I’m also interested in alternatives to AutomationIds. But I’ll need to wrap the standard asserts with something that positively checks asserts against requirement numbers.

Will the regulators go for this? I don’t know. When working in a regulated environment, you can be innovative with how you handle testing as long as the auditors are comfortable with your ability to be repeatable and cover what they find important.

3 Comments

Working with TestComplete, one of the first things we did was to make a CompareValues method that makes error logging consistent and outputs successes as well as failures. Outputting every little thing you check can be cumbersome at times, but when it comes time to review errors, it is also very helpful.

Regarding development variable names in code – there’s no way around that I think. If it were a manual test, you would be tied to the the text that shows up on the button instead of the name of the button. In my experience, there’s little difference. However, where automation wins big-time over manual procedures is in code reuse. In a manual procedure, you wouldn’t want to send the tester back to some page where you define the process to save a single file over and over again. But, in automation, you can define a procedure to save a file and use it over and over again (and test half a dozen points along the way, to ensure consistency in the application). My hope is, before long, you will have a slew of test helpers that let you do anything in the application you could want. Maintenance will be as simple as debugging one test helper. Writing tests will be as simple as chaining a few helper methods together. Getting there is a lot of work, but well worth it.

The biggest problem with automating system testing that I’ve encountered hasn’t been performance or maintenance, but state (though state can be tied to maintenance). Unit tests are easy and quick because your state is usually one or two objects – a relatively small chunk of memory that can be easily disposed and recreated. In system testing, your state is the entire machine. Resetting the entire machine is often not easy or even possible. What’s more, a good system test shouldn’t count on the machine being in a consistent state. It’s the border cases where the machine is busy getting a Windows update or something equally strange where applications tend to fail miserably. Writing your tests to handle every version of the application successfully executing an operation can be extremely challenging. I’m not sure what a test would look like which handles this gracefully. Perhaps some sort of “optional” flag that can be passed to a method like ConfirmOverwriteDialog so that, just in case the file already exists, your test will go ahead and save over it. There’s definitely some room for high-level design with tests.

Working with Ranorex allows you to implement your test with .NET languages. In addition you’re able to seperate identification information from test automation code. Hence, the typical maintenance task is outsourced. Ranorex works with xpath expressions containing structure and technology depending attributes to identify the used controls in the application under test.

I’ve recently heard of Ranorex. We were reviewing the tools my client was using and V&V in Miami really liked Ranorex, because of it’s use of C#. The catch he had was how it handled custom controls. He claimed it didn’t handle them to his satisfaction. On the plus side he could annotate his code and automate sections of a manual test runner at a time.

We did share how to use Functions as variables and first class functions in javascript to achive similar goals in Test Complete.

Custom controls are a big part of the WPF draw, and something I hope to address in a couple of posts.

Ranorex does intrigue me and I’ll have to grab a trial license once I have a better idea of what makes a testing tool the bees knees. It is about 1/2 the cost of TestComplete and if it has all the features I need, then great!

About Brian Ball

I'm a Software Engineer in Indianapolis. I'm interested in programming languages. I love dynamic languages like Ruby and Perl. I'm exploring functional languages like Haskell, F#, Clojure, and Erlang. I also use the .NET framework and occasionally deal with System Testing.

If you're not careful, you might also see some sketches and other art come up every once in a while.