Customer Acceptance Testing

I love testing, I could hardly wait for this meeting of the Chicago Ruby Group. I was not disappointed. Unit tests are cool, but specs are awesome. Whats the difference you say? I think its a more natural way to write your tests, it makes you think of the behaviour of your object and not "oh gosh, I have to write 3 tests for each of my methods."

On quite a few of my projects, I need people other than myself to test it. Projects that are more complex than simply displaying a list of phone numbers. Perhaps I have not looked in the right places, but I have not found much in the way of methods of getting people to test applications. I'll explain some of the ways I've tried and their result.

I've been working on a project for one my biggest customers now for many months. The system is huge, has automated a huge amount of work previously done by hand, and quite a bit on its own depending on specific business rules. One of the things it does is pull together various bits and pieces of information and puts it together into a single item for further processing.

The difficulty has been that there are so many rules, so many different conditions, and a variety of other things that affect the output, it gets to be quite difficult to test and preview the items. A few weeks ago, I had an idea... AJAX.

Back in college I was a math tutor. A preliminary for all tutors at this particular college was a joyous seminar on the “problems of learning and instruction” – a brief glimpse into just how maddening the enterprise of explaining things to people can be. Most of it seemed like tedium, and painfully obvious. It was no surprise to the assembled tutors-to-be that some people just don’t get math. Of course – why else would someone seek out tutoring?

I remember nursing a genuine resentment at being forced to waste two weekends listening to lecturers drone on and on about ‘cognitive gaps’ and ‘leaps of inference’ and the like when it was eminently clear that none of this would aid us, the tutors, or our charges. Really, if years all of this pinheaded theorizing hadn’t made oh, say, the professors of math courses any more effective as teachers, what chance did four days of it have for us?

Then came a little ‘interactive’ session entitled ‘How to Make a Peanut Butter and Jelly Sandwich’. The premise was simple. The audience was to verbally instruct one of the seminar’s facilitators, step-by-step, on the process of crafting a fine, and presumably edible, PB&J. He had everything he needed arranged on a table: peanut butter, jelly, a loaf of wonder bread, and a knife. Oh, dear, thought I. Now we get the sad spectacle of a man pretending to be as stupid as a bag of hammers in order to frustrate our attempt to walk him through making lunch.

And sure enough, the facilitator played it to the hilt. What was surprising was that this little exercise was actually enjoyable; hilarious even, as it became clear that this guy was experienced. He was good at what he did, and it made for fine comedy.

“Just start with two pieces of bread!” was the first instruction that coalesced out of the din of tutors offering simultaneous advice. The facilitator put on his best what-me-worry face and confidently gripped the bag of bread, held it aloft. Then consternation crossed his visage, signaling that the next phase of his mission wasn’t clear. Hmm… two pieces. Well, I’ve got this collection of bread pieces… hmm… I need two of them, but they’re in the bag… hmm.

“Open the bag!” Well, you lob a slowball like that, he has to hit it out of the park. A look of enlightenment came upon him as he vigorously shook the bag, but this, too, passed into confusion as the bag failed to yield its load of bread-like styrofoam. Much hooting and hollering ensued. Again, gripped by inspiration, he settled on a tried-and-true bag-opening tactic: he tore it open, violently. With purpose. With conviction. Bread flew from the bag. Some landed in his vicinity and he seized it triumphantly, mangling it further in the process.

“Now put some peanut butter on the bread!” At this point everyone was fully cognizant that this guy was going to misinterpret our instructions in some absurd manner; some of us just wanted to see exactly how absurd. Much laughter as he placed the whole unopened jar of Skippy on top of the bread with a self-satisfied flourish.

It went on and on; full minutes later, after we’d successfully negotiated the opening of the peanut butter jar, we’d told him to get some peanut butter on the bread by way of first getting peanut butter on the knife and then smearing the knife on the bread. See, we thought we’d out-dumbed him, but such was not the case as he viciously plunged the knife into the jar, punching out the bottom and sending a mess of broken glass and hydrogenated oils to the floor.

Well, I’ve bored you enough with the details. Suffice it to say that it took a solid hour to finally get this guy to make a frickin’ sammich. So where am I going with this, and what does it have to do with requirements gathering? Everything.

Because, when you’re writing software, you’re “sitting in a chair telling a box what to do,” as someone on JOS put it, and that box is pretty dumb. Because, essentially (and particularly in a business domain) you’re trying to instruct something a lot dumber than a man pretending to be stupid, and you’re trying to get it to do something a lot more complicated than making a PB&J. Another way of putting it is that successfully developing software necessitates dealing with a level of specificity that makes most people ill, insane, or both. Largely, this is a war of wills, and the decisive battle is usually in the requirements gathering phase.

As developers, we’ve probably all had the experience of requirements coming down from on high only to find that they are woefully incomplete and/or vague. That’s an old complaint and an old story. But it’s also a safe bet that we’ve all had the additional experience of running into what can charitably be called resistance when seeking clarification on requirements, and smoothing that dirt road is what I’m here to talk about.

Many approaches over the years have been offered. Agile/XP-type processes, with their emphasis on “user stories”, short cycles, and lots of end-user feedback, seem to work very well when they work, but, just like technologies, methodologies don’t exist in a vacuum. You usually can’t simply combine a methodology with a business environment like you combine acids and bases in a laboratory. Agile methods are great when everyone plays along, not so great when they don’t. Same goes for any methodology, big-M or small. The chief problem in any software development environment is cooperation—a people problem.

As developers, we’re quite aware of our strange mental inclinations. Whatever our differences as human beings, our similarity lies in our shared capacity to decompose the real world into a series of maddeningly specific steps. To a client’s business analyst, it’s a job well done to specify:
“Requirement 357a: The system shall, upon encountering an incoming address, increase the ‘returned from post office counter’ in the data warehouse for existing addresses equal to the incoming address.”

Yup, the analyst thinks, that’s all there is to it. Problem is, of course, the developer can’t compile that sentence into workable code. Now the developer is faced with the task of asking The Stupid Questions. Incoming address? Incoming from where? Where do we find it? How are two addresses reckoned to be “equal”? So on and so forth. The business analyst starts to get exasperated; he’s got, like, four tons of this crap to go through, and he’s not a mind reader either. Now he’s got to go directly to the client and ask The Stupid Questions, because come to think of it he’s not exactly sure of how the client determines when two addresses are equal—OK, they’ve got the same street, number, city and zip, but this one address is missing a state/province code… seems straightforward, they’re still equal, right? ‘Cos if you have the zip you know the state… jeeze! But that box, that stupid, stupid box, doesn’t know that, and now the analyst has to ask what the client wants, which makes him look like a moron, because he’s paid to figure this junk out, and he hasn’t done it, or so the client seems to suggest every time he walks into her office with another list of questions… So he ignores the developer’s email and hopes they’ll just do something right for a change. And the developer sends more email and things get testy because now the schedule is slipping because there’s still these unimplemented features because the developer doesn’t want to code them until the requirements are clear since if she does and the client doesn’t like it then that will generate a bug report on her code, and too many of those look bad come review time. Now QA is getting testy, too (no pun intended) because how are they supposed to test unimplemented features?

Four weeks later, after the PM has called the VP to schedule a JAD session, it comes out that:
“Requirement 666a: The system shall consider two addresses equal when, and only when, at least the following fields in the incoming data source (defined in subpart J of definitions document Foo) are Unicode (see addendum 6) character-by-character matches on a one-to-one basis… [long and winding road inserted here]… Further as documented in the ‘null-field coalesceable’ specification, STATE/PROVINCE is not a required field for this process as the system shall normalize the city and state by the postal code, which is required…”

Welcome to the Dilbert zone.

So sure, you say, we’re all familiar with this kind of frustration. What do we do about it? Well, I have an idea. Keep in mind that’s all it is. I’m not selling any snake oil. There’s no guarantee that this will work, no statistically significant findings from a controlled study to back it up. But I think it’s worth trying:

Arrange a meeting with the client, and get them to tell you how to make a peanut butter and jelly sandwich.

Since we as developers are paid to systematize the world on behalf of other people, we have to do a better job of educating our clients on both the value and the pitfalls of what we do. As long as the rain keeps falling, no one knows or cares about our chanting and dancing around with the chicken bones. Come drought time, the mystery of our profession is our undoing. We’ve always been unfathomable pinheads, but in times of systemic failure we’re the unfathomable pinheads who failed. I think it’s possible, and desirable, to give people a sense of what we do. And if you get to clown around and make a mess in the process, why not?