A biology teacher has developed a rare and troubling neurologic
disorder. The signs and symptoms are well controlled with medications
but the side effects are unpleasant. He hears of an experimental
treatment that has produced impressive results. The treatment involves
surgically ablating selective parts of the brain at close proximity to
the brainstem. There is a risk that, during the procedure, vital parts
of the brain could be inadvertently damaged and the patient could become
paralyzed.

Our teacher decides to see the neurosurgeon, a pioneer in this
experimental procedure, in a big medical center. The good doctor
enthusiastically tells him that the procedure may bring about a cure.
The following conversation ensues:

Teacher: What is the risk of paralysis?

Surgeon: I can't really give you a figure. All I cart say is,
so far, 900 such procedures have been performed around the world, and no
patient has suffered paralysis. Yours truly has personally performed
200, and so far, no problem. Touch wood.

The good teacher is too polite to say how he really feels. While he
likes the prospect of a cure without medications, he is afraid of the
possibility of paralysis. He recalls a recent article (Dougherty et al.,
2009) on absolute and relative risks in The American Biology Teacher,
his favorite journal. After reviewing the excellent article, the teacher
has the following question: If zero paralysis has occurred after 900
cases, should the risk be calculated by dividing the number of paralysis
cases (zero) by the number of cases performed (900) (Dougherty et al.,
2009)? That would give a risk of zero! Simple logic suggests that it
cannot be zero. When doctors started doing this procedure, could they
have declared the risk to be zero after, say, 10 uncomplicated cases? By
the same token, just because 100 bombs have been diffused without one
going off prematurely does not mean that bomb disposal is always safe
work and all that cumbersome protective gear can be discarded.

Our biology teacher wants to know what the risk really is before he
makes up his mind about surgery. He has two choices. First, he can wait
until more cases have been recorded from around the world. The problem
is that his disease is rare, and he may have to wait for years before
enough data can be collected. That seems unacceptable. Second, he can
try to figure out the risk based on the 900 uncomplicated cases. He
starts with a literature search and uncovers three papers dealing with
risk estimation when the numerator is zero (Hanley et al., 1983; Eypasch
et al., 1995; Ho et al., 2000). The mathematics involved is somewhat
complex. Always on the lookout for interesting topics to teach, our good
teacher is determined to introduce this concept of risk assessment to
his students, but is wary that many of them may not have enough
mathematical aptitude (Dougherty et al., 2009) to understand the
analysis. Undaunted, he takes out his pen and paper and starts writing.

* Problem Solving

Let the true risk of paralysis with each operation be r. (Of
course, r is the million dollar unknown he wants to figure out.) The
probability of no paralysis of each operation must be 1 - r. What is the
probability of two operations without paralysis? Well, the chance has to
be [(1 - r).sup.2]. The chance of no paralysis after n procedures is [(1
- r).sup.n]. Since 900 uncomplicated cases have been performed
worldwide, n = 900. Using EXCEL, the teacher plots the chance of no
paralysis after 900 operations, [(1 - r).sup.n], on the vertical axis
against the risk of paralysis r on the horizontal axis, for r ranging
from, say, 0.0002 to 0.005 (Figure 1). Again, he does not know where r
lies on the horizontal axis. For all he knows, the true value of r may
even lie somewhere outside of this arbitrary range he has chosen.
However, he has to start somewhere.

Let's say for the moment that the true value of r, the million
dollar unknown, is very roughly 0.0008 (point A in the graph), meaning
that there should be roughly eight paralysis cases, on average, for
every 10,000 surgeries. (The number 8 is the product of 0.0008 x
10,000.) This translates to 0.8 case for every 1,000 surgeries, or
0.8/1000 x 900 = 0.72 paralysis for every 900 cases.

The graph says that at r = 0.0008, there is a roughly 50% chance of
no paralysis after 900 cases. In other words, the chance of a clean
record after 900 cases is, well, about the same as flipping a coin. One
could just as likely have at least one case of paralysis after 900 cases
and, suddenly, the reassurance given by the doctor, based on no
paralysis after 900 cases, is not so reassuring. Remember that we are
not talking about loosing a few strands of hair here; we are talking
about a serious complication.

Let's say for the moment that the true value of r is very
roughly 0.0033 (point B in the graph), meaning that, for every 10,000
surgeries, there should be roughly 0.0033 x 10,000, or 33 paralysis
cases, on average. This translates to about 3.3 cases for every 1,000
surgeries, or 3.3/1000 x 900=3 paralysis cases for every 900 surgeries.
According to the graph, the chance of zero complications after 900
surgeries is 0.05, or 5%. In spite of this low probability, that is
exactly what has happened--no paralysis after 900 cases. It is therefore
unlikely that the 0/900 record has merely been due to incredibly good
luck. In somewhat convoluted words, there is a 95% chance that it is not
luck that no paralysis has occurred after 900 surgeries. The good
teacher likes the implications of being able to say with 95% confidence
that the true risk of paralysis is no more than 0.0033.

[FIGURE 1 OMITTED]

It turns out that Rumke (1975) had determined the rule of thumb for
determining the 95% confidence limit of the risk of an adverse event
when the numerator for risk calculation is zero. It is 3/n, where n is
the number of cases performed. In our case, after 900 uncomplicated
cases, one can say with 95% confidence that the risk of paralysis should
be no more than 3/900, or, 0.0033.

The next question our teacher needs to ask is: What is his chance
of paralysis if he were to undergo surgery? In other words, what is the
chance that the 901st case in the world (his case) to be performed will
result in paralysis? The answer has already been determined above. He
can be 95% confident that the chance of paralysis with his surgery is no
more than 0.0033, or 1/300. He would like to have the risk at less than
0.0008, or 1/1250, but he is only 50% confident that this more
attractive "worst case scenario" risk estimate is accurate.
Notice that the right vertical axis in the graph (Figure 1) is the
confidence limit, and is the percentage form of 1 minus the
corresponding values on the left vertical axis. Reading off the graph,
our teacher can also say with 99% confidence that the chance that his
surgery will be complicated by paralysis is at worst 0.005, or 1/200.
One may notice by now that if our teacher should demand that the
accuracy of the estimated risk be stated with a very high degree of
confidence, then that estimated risk must necessarily become very
conservative. In other words, the estimated risk would be relatively
high and might indeed be an over-estimation. Such is the price to pay
for demanding a high level of confidence when estimating the risk of a
complication. The confidence limit one picks will depend in part on how
badly one wants the treatment relative to how serious the undesirable
complication in question is. If the potential complication is quite
severe and could potentially overshadow the benefits, one might wish to
gauge the risk by choosing a high confidence level such as 95%, or even
99%. If the potential complication is not that serious, especially when
considered in light of the potential gain from treatment, then one might
not be particularly worried and accept a risk estimate based on a more
pragmatic confidence level. Notice that it is unreasonable to demand a
risk estimate with absolute, i.e., 100%, certainty. No doctor can
promise with certainty that no complication will ever occur. With that
in mind, our beloved teacher can make a more informed decision as to
whether he wishes to go ahead with the surgery.

Scientists, engineers, and doctors are constantly developing new
technologies, techniques, and therapeutics to improve health care. The
Food and Drug Administration is extremely safety conscious on the one
hand. On the other hand, it has the mandate of approving new
therapeutics in a timely fashion so the public can benefit. In spite of
rigorous and stringent testing and lengthy trials, highly beneficial
therapeutics are sometimes approved before all plausible adverse effects
have necessarily emerged. Estimating the risk of a rare, but plausible,
serious complication when none has yet occurred is therefore an
important concept.

The excitement of new discoveries often fuels exuberance that
sometimes leads to loss of objectivity. For example, when
minimally-invasive removal of the diseased gallbladder was developed a
couple of decades ago, one of the theoretical concerns was injury to the
common bile duct by laparoscopic instruments. After several small series
of laparoscopic cholecystectomies without common bile duct injury,
surgeons could hardly wait to declare that the new technique had led to
no such injury (Eypasch et al., 1995). Now our biology teacher (and his
class) knows that does not mean zero risk. These days, laparoscopic
removal of many kinds of diseased organs, including the gallbladder, is
routine. However, as more data have accumulated, it has turned out that
common bile duct injury does occur and surgeons ignore this risk at
their own, and their patients', peril.

* Activity

1. Pretend that one of the students has to make the same decision
based on the same information used in this article.

2. Go through the logic as outlined above.

3. After assuming no complication after 900 cases, and going
through the reasoning, ask the students to plot a graph with multiple
curves representing n = 600, 900, 1200 cases. This would be a good
exercise to also explore the powerful graph-plotting functions of EXCEL.

4. Using the graph, determine the risk with 95% confidence for the
above values of n, and verify the 95% confidence level for risk
estimation is 3/n.

5. Plot more graphs for n equaling small numbers, e.g., 25, 30, 90,
and see if the rule of thumb of risk estimation with 95% confidence
equaling 3/n still holds at smaller values of n. The class will discover
that when n is 30, the rule of thumb 3/n for 95% confidence level has an
error of about 5%. When n becomes smaller, the rule of thumb produces
increasingly erroneous results. Those interested in estimating risks of
rare events for large and small values of n should refer to Newcombe et
al. (2000).

6. Ask the students to make the same argument for 90% confidence
and 99% confidence and come up with the respective rule of thumb
(answer: 2.3/n and 4.6/n, respectively). To get to the 99% confidence
level, teach the students how to adjust the scale of the graph in EXCEL
by double-clicking on the vertical axis and defining the minimum and
maximum at 0 and 0.01. This would allow easier determination of the
corresponding r values on the x-axis.

7. If the mathematics proficiency of the students allows, review
the solutions offered in Hanley et al. (1983) and Ho et al. (2000). The
formula for the rule of thumb is -ln[1-CL/100]/n, where In is the
natural logarithm and CL is the confidence level such as 90%, 95%, 99%
(Table 1).

8. The minimum value for n has already been determined for the 95%
confidence level. To determine the minimum value for n at which the rule
of thumb for 90% and 99% confidence level produces a [less than or equal
to] 5% error, the class can use various values of n surrounding 23 for
the 90%, and 46 for the 99% confidence levels, respectively. The results
are summarized in Table 1.

9. The more severe a complication, the more cautious one should
become. At this point, discuss when one might use the risk estimate with
90%, 95%, or 99% confidence.

* Conclusion

A zero numerator does not necessarily mean zero risk. It is
possible to estimate the risk of a rare plausible complication even when
it has not occurred, and when only the denominator is known. This is
done through evaluating the likelihood of n successful trials for
various confidence levels. The concept can be easily taught to
nonmathematics students using a simple graph.

Table 1. The rules of thumb for estimating the risk (r) of a
complication when no complication has occurred after n trials,
at various confidence limits; and the minimum number of trials
(n) without a complication to allow the use of the rules of thumb
for estimating r with error.
Confidence limits (%)
90 95 99
Estimated risk (r) 2.3/n 3/n 4.6/n
Minimum number of trials (n) to produce a <5% 23 30 46
error compared to graphs or calculations