Friday, November 28, 2008

I've been spending a good deal of time lately thinking about the cognitive process of testing: What makes a good test case? How do we "know" that the software is fit for use? And why is that an area of such little interest in many test communities?

One thing that can help are examples that we can draw rules of thumb from. The typical example is the triangle problem, where we have certain rules for types of triangles, an input for sides, and try to figure out if the software is calculating the correct type of triangle.

The triangle problem is a great example -- as long as you did well in Geometry in high school. I've been in search of other examples that don't require telling a scalene from a isosceles.

That said, you can test anything that consists of rules and examples. So here goes ...

Picture It: Maryland, 1982. You've been hired as an auditor to check to see if Weiss's Grocery Store in Frederick is enforcing the sale laws correctly. You turn on your Plymouth Horizon and leave Middletown, taking Route 40 east, over Braddock Heights - in the distance you can just barely make out the famous clustered spires of Frederick.

... and then you're turning into the Weiss's parking lot. Before entering the store, you grab your clip-board and review the rules:

E) Other items like toys, magazines, pencils, paper cooking supplies, and so on

The following laws apply:

1) Maryland State Sales Tax of 5% applies to all items except staples2) Only food items (staples and non-staples) can be sold on Sundays3) All times must have a white tag that lists the price of the item; that price must be correctly charged by the cahsier 4) Listed prices in the weekly 'sale' circular must be displayed with a yellow that; that price must be correctly charged by the cashier5) You must be twenty-one years of age to purchase alcoholic beverages; the cashier should card you if you appear to be under thirty-five6) Alcoholic Beverages cannot be sold between 10PM and 6AM7) You must be eighteen years of age to purchase cigarettes; the cashier should request identification if you appear to be under thirty

Your job is to ensure that the correct laws and taxes are applied, and items are sold for the correct amount. (If you really want to be testing software, imagine that the rules must be applied through self-service checkout stations.)

Challenge: How would you test this? Defend your answer.

UPDATE: Yes, I welcome clarifying questions, and NO, you don't need to test the website. I've linked to that solely as background information. The test is theoretical and set in 1982.

Wednesday, November 26, 2008

For that matter, here's a story I'd like to explore: On November 7th, we ate at Pizza Hut, and I had a problem with the boneless wings, in that they did not have any sauce on them.

That's right: Regular Buffalo wings have sauce; Boneless Buffalo Wings do not. They are like Chicken Nuggets - they have a breading. When I asked for sauce, the waitress said that this was not a "Wing Street" Pizza Hut, and they did not have sauce on the boneless wings.

That's right, according to this waitress, if you order boneless wings, you get something different things depending on which type of Pizza Hut you are in.

My offense was that these were "buffalo wings", and what made them such was buffalo sauce ... which these nuggets were not actually cooked in ... you get the point.

Yesterday, I tried to complain on PizzaHut.com and found that the complaint form was broken. To the tune of, when you click you get the complaint form, you get "error in / application."

I tried again today, got the form, filled in in exhaustive detail, and when I click submit ... nothing happens. It's possible something was transferred back to the server, but my form stays the same. No email that they got the message, nothing.

Looking around the website for a contact us brings me back to ... a form to fill out and send. No email address. No phone number.

Either these people are deliberately not interested in negative feedback, or they are in desperate need of a software tester. Probably both.

I remain unsatisfied. I may blog more on this after the American Holiday.

------

In other news:

There has been some interesting discussion lately on the agile-testing list, about if "good enough" software is something to strive for, or if we should strive for perfection. I won't rant on that right now (You can Google "Impossibility of Complete Testing" if you'd like, or just go buy Jerry Weinberg's new book Perfect Software and Other Illusions.)

Still, I will leave you with one little rule I use for software testing:

Heusser's Postulate: The better tester you are, the more test ideas you'll have, but you get no more time. So testing quickly becomes an exercise in risk management.

Thursday, November 20, 2008

Folks, I've got a ton of ideas to take Creative Chaos. I've been spending more and more time lately involved in discussions of KanBan - which is an evolving method that is slowly competing with Scrum.

I could write about the aspects of software quality that are hard to define requirements for - scalability, performance, reliability, usability - and what to do about it.

I could put more meat on the bones of "when should a test be automated?" (or "run unattended?")

I could talk about the business of software development, negotiating for nerds, or the dynamics of maintenance and legacy software.

Or I could just post more stories from my youth as a military cadet.

I'm probably most interested in exploring the idea of uncertainty. After all, we can't predict the future - yet we have these things called project plans with dates and deadlines and deliverables.

Those are just some of my the ideas bouncing around in my head. Still, at this point, after three years of evolution, you - the audience of Creative Chaos, might know better than me what to do next, because you know what appeals to you and why.

I covet your feedback, and promise to respond and take it seriously.

And, if you've ever just wanted a chance to heckle ... now's you chance.

Please consider leaving a comment for this post; it's your chance to steer the ship.

Wednesday, November 19, 2008

Then suddenly I noticed (shock!) that not only was the author a journalist, not a scientist, but he was actually an editor at Time Magazine, which has an editorial method in which editors write stories based on notes submitted by reporters (the reporters don't write their own stories), so it's practically designed to get everything wrong, to insure that, no matter how ignorant the reporters are on an issue, they'll find someone who knows even less to write the actual story. - From JoelonSoftware.com

In the article, Joel is knocking a style of journalism where you start with an interesting anecdote and use it to prove a point about something which you have no real expertise in.

But let's flip it around, and say a journalist was evaluating traditional development methods:

... which has a programming method in which programmers code stories based on notes written by designers that are based on requirements documents created by analysts that are assessments of what the customer actually wants. It's practically designed to get everything wrong, to insure that, no matter how ignorant the analysts and architects are on an issue, they'll find someone who knows even less to write the actual code ...

Yes, many shops to better than this. Yes, agile has helped. But before we throw stones, many development houses might be better off tending to our own knitting ...

But I thought I might add my $0.02 to today's article, and share them here: It might be better to paraphrase deming than to quote him:

"A good person, working in a system that incents him to bad behavior, will become a bad person eventually."

Example: Measure me by lines of code and bugs fixed and see what happens.

My experience with crappy agile is with companies that live in areas that lack a world-class CS school, that hire off the street, don't pay well, and provide a cubical-like existence with heavyweight policies and procedures. In this envrionment, it is very hard to attract and retain high-quality talent. What's worse, in some bureaucratic organizations, the incentives are more to be political than to excel at programming. It's no great surprise that these organizations struggled before, so they 'Adopt Agile', or "Go Agile", or whatever, but only take the easy pieces, and often insist on still having everything planned and scheduled up front. After a few projects, the business has seen a marginal productivity improvement and the programmers say things like "My job sucks less."

As always, the work you get out of it is a function of what you put in. For the work the people are willing to put into it, that's actually a decent return.

In other words, crappy Agile is a management problem. And while issuing nice Herman Miller chairs, a decent laptop, and paying just ever-so-slightly above market rates might not solve the problem - it sure will go a long way to help.

I've worked with organizations where management made a real effort to provide collaboration, choice, and some sense of appreciation for it's staff - and to work hard to make sure the organization was consistent - that it rewarded the right thing. I've worked with a few that didn't. It makes a world of difference.

"It seems very pretty," she said when she had finished it, "but it's rather hard to understand!" (You see she didn't like to confess even to herself, that she couldn't make it out at all.) "Somehow it seems to fill my head with ideas--only I don't exactly know what they are! However, somebody killed something: that's clear, at any rate---"

Of course, Jabberwocky is, more or less, pure nonsense - a bunch of buzzwords and fluff with a couple of actual English sentances thrown in. Just enough real words to make you feel as if there is something to "get."

Does that sound familiar?

If we search and replace a few key words, Alice could very well be talking about requirements documents from some projects I have worked on:

"It seems very interesting," she said when she had finished it, "but it's rather hard to understand!" (You see she didn't like to confess even to herself, that she couldn't make it out at all.) "Somehow it seems to fill my head with ideas--only I don't exactly know what they are! However, customers will be able to do some new things: that's clear, at any rate---"

Many project documents take this approach - especially one where there is little agreement on what should be really done. So we paper over disagreement with big words. (Example: Can't agree to use a touchscreen, mouse or keyboard? Call it "The Input Device." Can't agree on how to run the project or what metrics to use? Simply insist that the project be "Quantitatively Managed." Can't agree on what a good organization looks like? Call it "Mature" or maybe "Agile.")

However, we have social problems with admitting that something is wrong ...

A) We are afraid that the document is good, and we're the stupid one.

B) We are afraid that we'll look dumb; that when we say "I don't understand this", the other person will pat us on the head and tell us that someday, indeed, we'll get it, when we grow up,

C) The team has obviously invested considerable time and effort in developing these documents. Saying we might as well start over can be taken, as, well, an insult to the professionalism of whoever prepared this document.

I don't have any easy answer to this problem. I do know that calling out dead-fish communication in any form can help. (It doesn't have to be bad documentation, it could be hit-and-run management, for example.) I do know that after living with bad documentation once, if you call things out early, it is sometimes possible to prevent them on the next project. (It works like this: "I know project X is coming. I know Alan the Analyst is working on documentation in a corner. That's just what we did on the SuperWidget project. We don't want another superWidget, do we?")

Monday, November 17, 2008

Brian Marick recently did a keynote at Agile Practices '08. While we have no video, and, sadly, no audio, he has made the entire text of his keynote available on his blog. Check it out. It's some really good stuff that has needed to be said for years.

I like this, and it continues to align with my ideas about writing in development and testing: That most people talk about the way things "should be" in some sort of abstract sense, and that the best writers talk about what actually happens in software development and what you can do about it.

Wednesday, November 12, 2008

Years ago (decades ago?) I was a cadet, cadet officer, and later adult officer in the Civil Air Patrol, the US Air Force Auxillary. While my responsibilities have pushed out the CAP, I still belong to an affiliated site, CadetStuff.org, where I used to write a column titled "Leading The Way."

After a brief discussion on the forum, I'm considering publishing a new article on leadership. Just for grins, I thought I would post it here. It's not your usual Creative Chaos Fare, but I thought you might enjoy it. If you'd like to see more, or less, of this type of material, please let me know through comments!

//BEGIN ARTICLEPicture it, Fort George G. Meade - June, 1995. Maryland Summer Encampment Staff Selection and Training. The "top three" command staff was already selected. The rest of the potential staff - 20 or more cadets ranked cadet major to staff sergeant - were vying for the great prize of leading basic cadets in training.

It was a great, glorious day, which I remember like yesterday. I pulled up in the late 1980's-era blue station wagon on Friday afternoon, sauntered over to the sign-in table, said hello the cadet deputy commander and XO, and signed in.

Heusser was here and his goal was clear - to command a squadron. Ah, squadron command. So glorious that to this day I still wax poetic about it. The opportunity to do real indirect leadership but still have someone concrete to compare yourself to - the other guy, the other squadron commander, the enemy.

Ok, maybe "the enemy" was a bit much, but you get the point.

So I signed in. Friday night we didn't do much; people would trickle in all night and the official training schedule started Saturday. We mostly sat around, told "war stories", re-introduced each other, and studied memory work. Most of the troops had just graduated in 1994 and didn't know what to expect; I'd skipped 1994, commanded a flight in '93, and was one of the old hands - one of two cadet majors applying to be on staff.

Saturday morning we work up, put on our battle dress, and waited. Not too long later (0700? 0630? I'm not sure), the command staff walked in.

Something was wrong. They weren't looking at us like staff candidates - we were cadet basics all over again - or something. I couldn't put my finger on it.

"Why aren't you formed up?" asked the cadet commander, with an edge in her voice. Oh, it wasn't mean - I don't mean to imply that we were "hazed" or anything silly like that. She was upset.

Thinking to myself, I wondered why she thought we should be formed up, and where. I kept this to myself - as they say, it's better to be thought a fool than open your mouth and remove all doubt. Besides, I was sure she would tell us.

Sure enough, she did. The training schedule, with times and locations, was listed on the sign-in table. We hadn't noticed. We were starting this whole thing off on the wrong foot.

Now time can do some interesting things. In some ways, we forget, but in others, things that were confusing can become more clear as we have time to reflect.

Now, if Cadet Staff Sergeant Snuffy knew about the schedule and told me, I'd have been remiss, but I'd also have marched the troops to the assembly area. The reality was, no one - not one - of the 20+ candidates had noticed the training schedule.

It wasn't solely a problem with our attention to detail.

It was /also/ availability of information and communications. As much as it was a problem with the troops, It was also a command problem.

Of course, I didn't say anything at the time. I knew my goal - to command a squadron - and that bringing up a complaint about the behavior of senior leadership was not the best way to get that command.

In hindsight, I wish I had said something privately, but in the grand scheme of things, I had more than a few behavior problems myself at that age.

However, that Saturday was memorable to me for more than one reason. Because that afternoon we were having a discussion about how to run encampment. And, yes, the "tear 'em down, build 'em back up" school of thought was dominant, along with the "fail them on the first few inspections to make a point" school, and yes, the idea of fear and intimidation as the way to change behavior.

And, cliche as it was, I stood up against the crowd and said something very much like this: "I think there are other ways to lead cadets. After, all when we command cadets, shouldn't our goal be to inspire cheerful and willing obedience to orders?"

That one sentence changed the tone of the entire conversation.

Now I'm not perfect, and I don't claim to have done everything right that weekend, and at the encampment that followed, where yes, I did command a squadron.

And when I reflect on all the mistakes I have made over the years, In my mind's eye, I still see that weekend as bright and shining. Because on two occasions, I saw that leadership had the opportunity to blame and abuse others, or inspire them, and I saw and preferred the better.

When you lead cadets at meetings, and activities, and at summer encampment - what is your goal?

Thursday, November 06, 2008

Economic Times are tough indeed. Security is hard to find. Yet there are some people who seem to always succeed despite tough times - and no, I am not talking about slick Sam the salesman.

Now, Sam might have questionable ethics. We may not want to be like him, we may not like his ethics or technique - but what he does, on some level, is important. Sam solves a unique business problem: Get the cash flowing in.

I submit to you that if we can find ways to solve business problems (or create opportunities), we can avoid the attitude of "I've got fifty guys lined up around the block, you'll take what I offer and be happy" for businesses, even in tough times.