Where are the Failures?

We all learn from our mistakes, but how can we learn from them if we never
find out about them? Success is rewarded and failure is swept under the rug.
Fading failures are a pervasive part of today's culture. If there is success,
well and good, let people know about it, but today's mantra is that if success
is not achieved, declare success and move on to a new challenge. What manager is
willing to admit failure? This applies, unfortunately, to science as well as the
rest of life.

Peer reviewed journals rarely publish instructive failures. Project managers
of failed projects do not publish what went wrong or their assessment of their
failures and what could have been done better. If one were to judge based on
trade and marketing publications, one would think that there were no failures,
only a parade of improvements, minor or revolutionary. Examining journals, one
must conclude that research leads to a mass of minor advances. Failures are
hidden - how can one learn from them?

The lack of research and development failure analysis is a major problem,
primarily because it then becomes impossible to tell the reason for failure,
whether failure could be avoided, whether great successes lie hidden in the
broken pieces of a failed effort. Was the failure due to fundamental problems?
Incompetence? Innocent mistakes? Misguided direction? Lack of technical
developments in related fields? Torpedoed due to political jealousies?
Superceded by more successful parallel projects? Insufficiently funded? Funding
discontinued at a crucial juncture? Or perhaps the work is simply not
sophisticated enough. There are many more ways to do things wrong than there are
to do them right.

From the general to the specific. I am saddened by many failed projects that
I know of, many directions that will not be taken because they were tried,
failed, and for that reason are now deemed impossible. Quite often
inappropriately so. Failure is not an end, but only an obstacle sometimes to be
overcome, always to be learned from.

And again, how now to overcome this failure? How can past failures be brought
to light and used to contribute to progress? The answer is first to identify the
trails of advertised dramatic success that faded into oblivion, and then to find
the buried reports that discuss the problems being faced, trying to figure out
what happened and whether success was hidden in failure.

Unfortunately this is a time consuming process, rarely funded or appreciated,
and rarely rewarded. Even more, an analysis showing how to avoid the failure
rarely overcomes negative assessments from those who shepherded the project to
its failure, because these are precisely the people who would be shown lacking
on the failed project. Usually time must pass until these people move on with
their damageable reputations, replaced by others with less prejudice.