Blog: Oracles and The Right Answer

Tony’s brow furrowed as he spoke. “No oracle comes with a guarantee that it’s giving you the right answer. That’s what you said. But surely there are some oracles that are reliable,” he said. “What about pure math?”

“Pure math? All right. Here’s an example: what’s 61 plus 45?”

“Duh. 106.”

“Well,” I said, “for many computer systems prior to the year 2000, if you added 45 to the year 61, you’d get 6. That is, if you looked at a printout or a screen, you’d expect to see “06” in the year field. And for those systems, that would have been the right answer.”

“But that was wrong! Y2K was a problem. They called it ‘the Y2K problem‘, didn’t they?”

“True,” I said. “But until the late ’90s, it wasn’t a problem—or to be more accurate, people didn’t perceive it as a problem. On the contrary, it was a solution to a problem: memory and storage were expensive. You could work around the “problem” with a combination of clever code and trust that people would interpret the output appropriately. Remember, a problem is a problem to some person at some time. Programmers and designers in the 1960s had one set of problems to solve, and programmers at the end of the ’90s had another set. The point is that one oracle (regular math) would give you one right answer, and another oracle (what the programmers and designers wanted) would give you another. Listen: no oracle can give you the right answer. An oracle can give you a right answer—a plausible answer that might be right for its context. But changing the context can flip that right answer into a wrong one—or a wrong answer into a right one.”

“Oracles are heuristic,” I continued. “There’s this terrific book, Discussion of the Method, by Billy Vaughan Koen. He’s an engineer, but he’s also a philosopher of engineering. In the book, he makes the argument that all decision-making, all problem-solving is heuristic.”

Tony looked quizzical. “Wait… Even algorithms? ‘Algorithm’ is the opposite of ‘heuristic’—didn’t you say that?”

“Not exactly. Algorithms are robust; they tend to produce very reliable results. But Koen says that even algorithms are heuristic. After all, if you apply an algorithm in the wrong way, to solve the wrong problem, or in the wrong context, it will fail.”

“Aaargh,” Tony said. “Where does that leave us? How can we ever know when a program’s correct?”

“That’s the interesting part,” I said. “We can’t. A program can appear to be working in all kinds of ways, but the program and your oracles can fool you. Think of a calculator program. Yep: one plus one gives the answer ‘2’. That’s looks correct to you, right?”

“Sure.”

“And yet if the calculator is in binary mode, the answer should be ’10’. You might be applying the wrong oracle for a given problem. Even if the program isn’t in binary mode and ‘2’ is right, the program could be tying up the processor so your machine is unusable. Or the program gives you the right answer—in white text on a white background. Or the program clobbers the contents of the clipboard. And you don’t notice these things unless you’re looking for them, or unless you happen to notice them. That is, there might be a problem for which you don’t have an oracle..”

“So nothing can tell us that a program’s working right? We can’t ever tell whether a program is giving us the right answer?” Tony asked doubtfully. “That doesn’t sound… right.”

“Working right, yes, but only in the sense that it appears to be fulfilling some requirement to some degree. A right answer, yes, but the right answer only in context, and not a complete answer. Correctness is a human notion, and things are only correct in some context. As testers, we can’t know for sure the deep truth about any observation. Any right answer that we see in computer software is only right for now, this time, for some purpose, on this machine. We can’t reliably project our observations into the future. We can use an oracle to give us as a strong inference that the answer will be the same next time, but we don’t get a guarantee. What we see might be right based on what we’re observing, but there’s all this stuff that what we’re not observing too. Cem Kaner and Doug Hoffman describe that stuff really thoroughly. You’ve heard that complete testing is impossible, right?”

“Of course.”

“Well, part of that is the coverage problem; we can’t test every possible input to a program in a finite amount of time. But part of it is the oracle problem, too. We can’t see a problem unless we have an oracle for that problem: that is, a principle or mechanism for recognizing that problem. All our oracles are heuristic, fallible—and in software, the potential for problems is limitless.”

“So how do we get around that?” Tony asked.

“The first thing is to recognize that oracles don’t give us the right answer, but every oracle may be able to point us to some problem. Over the years, we we’ve studied oracles, we’ve come up with a bunch of principles and mechanisms for them, and we keep discovering more. Since there are infinite numbers of possible problems, we need a wide variety and diversity of oracles to spot them. But there is one principle that seems to prevail overall.”

I’v been scrutinizing the run rate, pass rate (don’t get me started on the expected pass rate) used in my client by all vendors and many onsite testers as well. My main points have been that fail doesn’t often mean failed, run doesn’t usually mean run, run percentage is more harmful than useful (mainly as it guides testing into wrong direction), and that passed doesn’t often mean passed. Thanks for giving more ideas to think about for those discussions!

And yea, I havent used those “metrics” in about a year At all anymore. We’ve been talking much more since, which all sides seem to enjoy.

Your email address will not be published. Required fields are marked *

Comment

Name *

Email *

Website

Notify me of follow-up comments by email.

Notify me of new posts by email.

Past Presentations

You can find an extensive list of presentations and courses that I've taught, including the slides and speaker notes for many of them, here.

Coming up—let's meet!

Check out my schedule, and drop me a line if you'd like to get together when I'm in your part of the world. If you'd like me to work with you and it doesn't look like I'm available, remember that my clients' schedules are subject to change, so mine is too. Finally, note that some conferences offer discounts—and even if they don't advertise it, maybe we can work something out.

February 28, 2019

Wherever you are!

A takeover of the Techwell Hub #testing channel. I'll be asking questions interactively all day. Find out more and sign up here.

March 6, 2019

Wherever you are!

An online webinar, Higher-Value Checking. This webinar is based on the session of the same name that I presented at EuroSTAR 2018.

April 1-4, 2019

San Francisco, California, USA

The Software Test Professionals Conference (STPCon). I'll be giving two one-day workshops: Critical Thinking for Testers, and a Rapid Introduction to Rapid Software Testing. Then there's a keynote talk on Testers as Their Own Worst Enemies, and a track talk on Rapid Software Testing in Agile Contexts. More information here.

XP 2019 - The 20th International Conference on Agile Software and Systems Development. I'll be offering a one-day workshop on Analysis for Agilists, and a talk on Refactoring the Agile Testing Quadrants.