PAGING PAULA GORDON - Comments, please?

Are you aware of all the work that was done to make him believe this
to be true? Or is he just trying to calm us down?

NOVAK: Senator Bennett, there are an estimated 50 billion
embedded chips in the world, at -- I've read that six percent of them
may be defective, but because of the Y2K, but if only one percent are
bolluxed up, how much of a problem would that be for a variety of
things not working?

BENNETT: Yes, one percent would be an enormous disaster. And
when I first got into this the most responsible estimates I had were
two to three percent failure. There are some folks that have said as
six percent. But I don't really think it was ever that high.

We are now being told by the chip makers, and I've spent a lot of time
on this, that -- two things:

Number one the figure is too high. The real figure for failure
is going to be something like two-tenths of one percent,
instead of one percent or two percent.

And secondly, that there's a very real difference between failure and
failure. [Depends on what the definition of "is" is!] In other
words the chip can fail, but the device that it's connected to can
continue to operate because many times the date function is non-
essential. So, we've spent a lot of time worrying about that, checking
on it.

I can give you some horror stories because we know the horror stories.
But it now appears that this is not going to be as bad a problem as we
first thought . . . .

Lets see, I've got ten fingers. One percent would be one finger. In the event I lose that one percent, I won't operate as I did, right? Not to mention the fact that during the time I am adjusting to the loss, I won't be functioning well. In fact for a time I might not be functioning at all!

I seriously wonder if some of these people know the difference between a computer chip and a "chip" in Las Vegas.

I suspect it may be a serious error to assume that
anyone in government or out of it knows what's going to
happen on and after the rollover. Or even the degree of disruption to
plan for. Like most of us, most of "them" are human, are not
clairvoyant, are not noted for extraordinary wisdom, and are unable
to foretell future events.

Anyone can see, and say, what might happen. We've been doing
that here for months now.

I noticed that Senator Bennett had changed his perspective on
embedded when he gave a talk at an IQPC Workshop in Arlington,
Virginia on March 17 of this year. I could not believe what I heard
him say and indicated that in the Q & A, challenging him to explain
the grounds for his statements. He indicated that he was basing what
he was saying on what he was being told by corporations. I spoke with
him after the talk and urged him to take another look at the sources
of his information as well as his conclusions. I urged him to focus
on the highest hazards sectors where the stakes are far higher and the
percentage of potential failures far higher as well. He said that he
was taking a close look at the chemical sector (he had indicated so
such concern in the presentation he had just made).

The following are various thoughts regarding Senator Bennett's
apparent perspectives on embedded systems.

Senator Bennett does not appear to have kept up with the potential sea
change in understanding that has taken place since November 9, 1999 at
the President's Council on embedded issues. But, then even the
President's Council does not seem to fully comprehend the implications
of the November 9 meeting with embedded systems experts and the
subsequent press release by the Secretary of Commerce, the posting of
an article on the subject at the NIST website by Gary Fisher of NIST
and Michael Cherry of Century Corporation, and the statement issued by
John Koskinen that can be found on other threads on this forum.
Perhaps, Senator Bennett has not studied the results of the
November 9 meeting. Perhaps, no one has brought them to his
attention. It may also be that there are opposing views on the
subject among his senior Committee staff. It may also be that the
Senator has not been fully briefed concerning the opposing points of
view either within the Committee staff or outside.

Aside from the possibility of being ill-informed or uninformed,
another way of explaining Senator Bennett's change in perspective is
simply that he has succumbed to wishful thinking.

Another way of explaining the Senator's change in perspective is that
he has never fully comprehended the nature, scope, and seriousness of
the embedded problem.

Another way of explaining the change is that he has adopted the
mainstream inclination to focus on probabilities rather than stakes.
He apparently is minimizing the seriousness of millions of embedded
systems malfunctioning, some of which will have immediate or near term
consequences and others of which will have with longer term
consequences. Those who take the "probabilities" side of the "stakes"
vs "probabilities" argument typically minimize the seriousness of
malfunctioning systems. They typically minimize the implications of
such malfunctioning and failures for public health and safety, the
sustainability of the environment, and the integrity of the social
fabric. Values come into play here, as well as understanding,
knowledge, experience, and common sense.

Denial can be playing a role in his change of perspective. One
Chernobyl in the U.S. would have horrific consequences. Indeed, Three
Mile Island, had it proved catastrophic in 1979, could have had
devastating consequences affecting an area that could well have
included the nation's capitol. Several Bhopals or even mini-Bhopals
scattered here and there would also be unthinkable. The mere
recognition that such catastrophes are possible here as well as in
other parts of the world, can have psychologically paralyzing effects
on those who recognize the possibilities. This can include persons in
roles of public responsibility. It can be easier for a public
official to take the path of least resistance and simply try to
convince oneself that it cannot happen here. In a word, denial can be
the easiest way out.

In my conversation with Senator Bennett in March, when he indicated
that he had changed his perspective based on the information that he
had been getting from corporate contacts, I told him that the
percentages of failure were sector specific. He seemed not to be
aware of that. I told him that I would be happy to meet with him and
his staff and share information with him. I also said I would send
him another copy of my White Paper since he seemed not to have seen
the copy I had provided his office in February. I had met before and
after with members of his staff and I continue to raise questions of
him in public meetings.

Another reason for his change of perspective could well be a small "p"
"political" one: those in political roles who feel that they may have
gotten too far out on a limb and wish to be closer to the mainstream
can rectify that situation by backtracking and distancing themselves
from previous statements and perspectives. This certainly seems to be
the case with Senator Bennett. In his most recent appearance on Y2K in
September at the National Press Club, Senator Bennett even renounced
and belittled his own formerly held perspectives. I have reason to
believe that his wish to align himself more with the mainstream has
played a major role in his about face. Standing up for what one
believes to be true seems to be an increasingly rare attribute among
those in public life. Standing up for what one believes in is an
attribute of leaders who exhibit statesmanship. Senator Bennett
himself had exhibited such statesmanship in his landmark speech on Y2K
on July 15, 1998 at the National Press Club. Senator Bennett, of
course, is not the first to have backtracked in this way. Social,
psychological, and political pressures can cause many to change their
convictions. When a political figure is also uninformed or
ill-informed about a complicated issue, it far more likely that that
individual will succumb to such pressures.

It should also be noted that the most recent reports and efforts of
the Senate Committee have continued to include a range of damning
findings involving embedded problems. This can been seen in the
final report of the Committee and in the intermittent concerns of the
Committee for the chemical sector (dating particularly from around
February of 1999) and since early October 1999 for nuclear power
plants. Senator Bennett has alluded publicly to such concerns, even
held hearings. He has spoken of instances of failures and expected
failures. When speaking in sound bites, he appears to either be
forgetting what he knows, ignoring its significance, or otherwise
minimizing its importance.

Perhaps, his statements simply reflect the ruminations of an
individual who has tired of the subject and is worn down by the
daunting nature of all the problems that he and his Committee have
uncovered.

It seems very difficult for the Senator to connect the heartfelt
concerns that were especially apparent last year with his subsequent
intermittent concerns and more recently awakened interest in chemical
sector problems and possible nuclear sector problems. He appears not
to be able to come up with a coherent perspective on these matters.
It is as if he were several different people making statements which
were in no way consistent with each other now or over time.
It is not always possible when one speaks in soundbites during media
appearance to reflect the breadth of concerns that one has. It would
be helpful if he were interviewed by someone who was aware of his
changing perspectives and the inconsistencies in his statements and in
the findings of the Committee. It would be helpful if he were
interviewed by someone who could call upon him to reconcile the
contradictory nature of what he is on the record for doing and saying.

Perhaps he is not aware of the inconsistencies. Or perhaps he is
aware of the inconsistencies and he does not think it important that
he try to clarify his thinking or his pronouncements. Perhaps, he has
tried and not been able to reconcile the inconsistencies and he is
simply putting the best face on the problem, so as not to panic the
public.

I take it from the direction you came at the question, that you don't
personally believe there is any basis at all for Bennett's more
optimistic veiwpoint?

Also, I have seen alot mentioined about this "sea change" in the
white house thinking on embeddeds because of a November meeting.
Anywhere we can get more information or discussion what that meeting
was all about and some of the results?

Or is the best we have suppostion because of some of the change in
public pronouncements?

"I take it from the direction you came at the question, that you don't
personally believe there is any basis at all for Bennett's more
optimistic viewpoint?"

Not an iota. In fact, the statement by the Secretary of Commerce, the
NIST material, and John Koskinen's subsequent statement in late
November would likely raise anyone's concerns who was rightward of a 7
on the impact scale.

You also wrote:

"Also, I have seen alot mentioned about this "sea change" in the white
house thinking on embeddeds because of a November meeting. Anywhere we
can get more information or discussion what that meeting was all about
and some of the results?"

I will attach an excerpt from other writing I have done on that
subject at the end of this message, plus a copy of the statement that
John Koskinen offered in late November. I would not say that
these development have influenced White House thinking. Mr.
Koskinen's views have been changed and some members of the Council.
I don't see any evidence that the President knows about the statement
or, if he does know about it, that he understands its implications.

...On November 9, 1999, the Presidents Council and the Office of
Management and Budget convened a meeting involving a small group of
embedded systems experts. The result of that meeting was reflected in
part in a press release that was issued by the Secretary of the
Department of Commerce. On the same date, the National Institute of
Standards and Technology issued an article that focused on embedded
systems issues. The Secretary of Commerce urged that efforts need to
be redoubled to test for year 2000 computer problems that are hidden
away in a variety of machines other than computers. See
http://www.nist.gov/y2k/embeddedarticle.htm and
http://www.nist.gov/public_affairs/releases/g99-204.htm
The Chairman of the Presidents Council was questioned about the
November 9 meeting at the Press Briefing held on the occasion of the
release of the Councils Final Assessment Report at the National Press
Club on November 10. A New York Times reporter wrote the following of
the exchange that he had with Mr. Koskinen after the formal Press
Briefing had concluded.

'Another concern, which Koskinen said he was briefed about on
Tuesday at an Office of Management and Budget meeting with computer
specialists, is that some computer systems that do not appear to
track the date may nonetheless have date-sensitive microchips in
them. Those systems also have to be tested and plans must be made
to handle breakdowns, Koskinen said.' From:
http://www.nytimes.com/library/tech/99/11/biztech/articles/11year.html

According to an embedded systems expert who is acquainted with Mr.
Koskinens change in perspective on this issue combined with my own
knowledge of what was determined at the November 9 meeting, the quote
should more correctly have read (needed changes indicated in all
caps):

'Another concern, which Koskinen said he was briefed about on
Tuesday at an Office of Management and Budget meeting with EMBEDDED
SYSTEMS specialists, is that some EMBEDDED systems that do not
appear to track the date may nonetheless have date-sensitive
microchips in them. Those systems also have to be tested and plans
must be made to handle breakdowns, Koskinen said.'

I would add these major and continuing concerns regarding embedded
systems failures. The first is from my Part 2 of my White Paper:

When embedded systems fail, they can fail in a variety of
unpredictable ways. Small, seemingly insignificant failures can
trigger other system failures." [From Page 40 of Part 2 of my White
Paper: "A Call to Action: National and Global Implications of the
Year 2000 and Embedded Systems Crisis". See
http://www.gwu.edu/~y2k/keypeople/gordon.]

I would also add that the timing of the triggering of other system
failures cannot be readily predicted since the environment in which
the failures are taking place is dynamically changing. Once the
failures have occurred and have triggered other failures, the root
causes of the initial failure can be hard if not impossible to
determine.

Understanding embedded systems is crucial to understanding the crisis
nature of the situation that we are in. The absence of understanding
of embedded systems has played a major role in the governments
approach to addressing Y2K. In my view, the failure of the
Administration to recognize from the outset the importance of
consequences of the malfunctioning of embedded systems has resulted in
an extremely flawed approach to addressing the problem and a failure
understand its complexities, along with a failure to recognize the
crisis nature of the problem.

The Presidents Council has failed to give adequate attention to the
highest risk, highest hazard systems, plants, sites, pipelines,
facilities, etc. The Presidents Council has failed to take the
action that it should have taken to help ensure that impacts that can
be expected as a result of malfunctioning embedded systems in highest
hazard, highest risk sites, plants, facilities, systems, pipelines,
refineries, etc., etc. would be minimized to the extent humanly
possible.

Even with the late recognition concerning the seriousness of embedded
systems problems as of the November 9 meeting, no major initiatives
involving embedded systems have been apparent on the part of any
agencies or departments of the Federal government apart from the
statement of the Secretary of the Department of Commerce. The
important implications of the November 9 meeting and the subsequent
press release and article at the NIST website, seem not to have been
recognized or shared with the President, the Secretary of
Agriculture or the Secretary of Energy, based on remarks they have
made since the November 9 meeting.

Attachment 2:

John Koskinen's Statement Regarding Embedded Systems and
His Perspective Concerning the Results of the November 9th Meeting

[I have added my comments in all caps in the text of the list of
"final statements". I have also numbered the "final statements".
They are not numbered in the original text.]

PRESIDENT9S COUNCIL ON YEAR 2000 CONVERSIONS
MEETING ON Y2K EMBEDDED SYSTEMS
Tuesday, November 9, 1999
American Society of Association Executives Building
1575 I Street, Washington, DC

(Statement issued by John Koskinen circa 11/29/99)

Participants in the meeting included technicians that had done work in
the bio-medical, defense, electric power, gas, manufacturing, oil,
shipping, and telecommunications industries. To help with the
discussion, an agenda was provided with discussion statements
concerning the types of embedded systems potentially atY2K risk,
difficulties in testing for such embedded systems and fixes for
problems found. Those statements were revised during the meeting and
the agreed upon final statements are presented below, along with
a brief summary of the discussion that led to the final statement.

Types of embedded systems found to have a Y2K risk:

[1] Final Statement: Embedded systems are at risk of problems during
Y2K rollover if they conduct a calculation that depends on a
representation of the date. The date could be in "relative" or
"absolute" form.

The participants presented a number of specific cases where they had
found Y2K problems in embedded systems. Several of these involve
calculations of time increments inside an embedded system without the
date being displayed or apparently used. In these instances an
embedded system calculates the time interval by subtracting seconds
from seconds, minutes from minutes, hours from hours, and calendar
dates from calendar dates.

All except one of the examples were large, complex processes where
embedded systems inter-relate with each other and, in some cases, with
external computer systems. The one example was of a stand-alone
embedded system that was unconnected to others that did not apparently
involve dates. That example lead to a discussion about the need for a
continuous power source being available for any such devices to
function, and it was pointed out that in some sectors there are many
such devices, but that few problems had been found in them.

There was considerable discussion of potential failure rates of
embedded systems. Estimates ranged from a 1 - 2% potential failure
rate of processes containing embedded systems in some sectors to 4 -
6% in others, but no conclusion was reached.

COMMENT: IN SOME SECTORS, THE POTENTIAL FAILURE RATE IS FAR HIGHER.
SOME OF THOSE SECTORS INCLUDE THE SECTORS THAT POSE THE GREATEST RISK
TO LIFE, PUBLIC HEALTH AND SAFETY, AND ENVIRONMENTAL SUSTAINABILITY.
PG

......An important distinction was made between failure
of an embedded system, which may not cause a process or device to fail
in operation, and failure of a process or device due to an embedded
system. The former represents the estimates above, and the latter is
much less prevalent.

The remainder of the discussion during the meeting focussed on large,
complex processes that contain embedded systems. The question of
having a real time clock or access to a clock was discussed and
examples were presented where the time was set by a process controller
and transmitted to other embedded processors involved in the process.
Other examples of problems were discussed where time was used
apparently to calculate relative increments (e.g. day of the week) as
opposed to absolute dates.

When embedded systems will fail:

[2] Final Statement: Where possible, all mission critical systems
should be tested end-to-end, whether or not the systems appear to have
date sensitive functions. Failure to do so means a small level of
risk has been assumed that, at minimum, should be addressed with a
contingency plan.

COMMENT: THE USE OF "SMALL" TO CHARACTERIZE THE LEVEL OF RISK
MINIMIZES THE SERIOUSNESS OF THE FACT THAT THERE IS RISK. THE RISK
MAY INCLUDE SOME HIGHLY SENSITIVE "SAFETY CRITICAL" SYSTEMS IN NUCLEAR
POWER PLANTS, CHEMICAL FACILITIES, REFINERIES, HAZARDOUS MATERIALS
SITES OR FACILITIES, PIPELINES, WATER SUPPLY SYSTEMS, AND SEWERAGE
DISPOSAL PLANTS. PG

The discussion that lead to this statement began with a presumption
that embedded systems involved in calculating time increments, as well
as those that apparently computed dates, are at Y2K risk. During the
discussion the statement to "test mission critical systems whether
they have a date function or not" was almost agreed to, until it was
pointed out one can only test those types of devices with end-to-end
testing.

This statement was focussed on mission critical systems because it is
difficult and expensive to conduct such testing. The term mission
critical systems was used to include safety critical systems as well
as other systems where the cost of failure would be high. Therefore,
while the statement says the risk of failure is low, the impact of any
such failure would be high.

COMMENTS: THIS GETS US INTO THE STAKES VS PROBABILITIES ARGUMENT. THE
STATEMENT REFLECTS A FOCUS ON THE PROBABILITIES SIDE OF THE ARGUMENT.
PG

...The statement also recommends a contingency plan to help mitigate
risk -- such a plan should not be viewed as an alternative to testing
because detection of a failure may be difficult and a failure could
cause substantial collateral damage before it is detected.

[3] Final Statement: The majority of failures of embedded systems are
expected to occur on or about December 31st through January 1st.
However, simply turning a system off during that time frame is
generally not a solution.

COMMENT: THIS STATEMENT SEEMS TO ME TO BE HIGHLY QUESTIONABLE AND ALSO
MISLEADING. THIS STATEMENT DOES NOT REFLECT AN UNDERSTANDING OF

~ THE FACT THAT THE TIMING OF FAILURES IS FAR MORE COMPLICATED THAN
THAT (FOR INSTANCE, SEE MARK FRAUTSCHI'S PAPER ON EMBEDDED SYSTEMS.
SOME ASSESSMENTS, INCLUDING GARTNER GROUP ASSESSMENTS, HAVE PROJECTED
PROBLEMS OVER TIME, NOT ALL AT ONE TIME.)

~ THE RESULTS OF FAILURES, INCLUDING MULTIPLE CASCADING FAILURES, WILL
NOT NECESSARILY BE IMMEDIATELY APPARENT.

~ THERE COULD INDEED BE ROLLING WAVES OF DISRUPTIONS AND DISASTERS
THAT GO ON FOR MONTHS OR YEARS OWING TO EMBEDDED SYSTEMS FAILURES THAT
OCCUR ON TOP OF IT-RELATED DISRUPTIONS. IT MAY BE IMPOSSIBLE TO TRACE
THE ORIGIN OR THE SEQUENCE OF THE FAILURES AND DISRUPTIONS AND ALL THE
MORE DIFFICULT TO UNDERSTAND HOW TO PROCEED WITH REPAIRS. PG

The discussion explored the question of whether the time of primary
risk of failure was during the rollover time. It was generally agreed
that the vast majority of failures in embedded systems are likely to
occur over that period. On the specific question of whether Greenwich
Mean Time would be a time of high failure, it was stated that most
failures would likely occur at 12:00 local time, although some would
also occur on Greenwich time.

During the discussion, there was a concern raised that the statement
may lead to the ineffective solution of turning off systems during the
rollover period. Therefore, the specific admonition not to rely on
that work-around was included in the statement.

[4] Final Statement: One can have two apparently identical systems of
which one will not have a Y2K problem but the other will have
operating difficulties. However, the chances of this are small.

The likelihood of failure of one of two identical systems, as
described in this statement was considered to be very small, but,
again, it was agreed that all mission critical systems needed to be
tested.

Difficulties in testing for embedded systems at risk:

[5] Final Statement: Organizations that have relied on a device
manufacturer9s declaration of Y2K compliance are at risk if they do
not keep up with the most recent manufacturers9 statements.

COMMENT: VENDOR CERTIFICATION IS OF QUESTIONABLE SIGNIFICANCE IF AN
EMBEDDED SYSTEMS IS LINKED TO ANOTHER SYSTEM OR SYSTEMS THAT ARE NOT
COMPLIANT. PG

The discussion concerned cases where testing had brought into question
manufacturers9 statements of the readiness of their products. A
number of instances were cited where problems had been found both
externally by users that had tested and by manufacturers themselves.
While the changes needed to remedy such problems have normally been
made quickly available, the concern was expressed that many
organizations were not aware of or taking advantage of those fixes.

[6] Final Statement: Some interconnection problems among embedded
systems can only be revealed by end-to-end testing.

The discussion concerned how to test for problems in embedded systems.
There was considerable discussion of difficulties of testing in
operational environments and the risks and complexities of end-to-end
testing. However, a number of examples were cited to show that one
could not find all potential problems in complex, interconnected
embedded processes without end-to-end testing.

Fixes:

[7] Final Statement: Anyone taking a fix-on-failure approach for Y2K,
particularly with embedded systems, runs a significant risk of
collateral damage and a difficult recovery.

There was little discussion leading to this statement. Remedying the
kinds of Y2K problems participants had found in embedded systems was
difficult and time-consuming.

[8] Final Statement: After a full and careful technical assessment,
there may be administrative or operational workarounds to many Y2K
problems involving embedded systems.

COMMENT: IT WOULD BE USEFUL FOR ANY FUTURE ITERATIONS OF THESE
STATEMENTS TO INCLUDE AT LEAST SOME REFERENCES INVOLVING THE HIGH RISK
EXAMPLES THAT WERE IDENTIFIED.

While simply turning a system off during the rollover is not normally
an effective administrative work-around, in some instance it could be.
Similarly, setting the year back so that Y2K does not occur may be a
work-around in some instances. However, before using these or any
other ways to work-around the Y2K problem, all agreed that a thorough
assessment of the full implications of the work-around was necessary.

[9] Final Statement: Even those that have conducted thorough testing
need to develop contingency plans for mission critical processes and
exercise them.

There was little discussion of this statement, in light of the earlier
statements that indicate the risk of Y2K problems.

COMMENT: THE IMPLICATIONS OF FAILURES AND THE NEED FOR NOT JUST
DEVELOPING AND IMPLEMENTING CONTINGENCY PLANS, BUT DOING THE SAME FOR
RESPONSE AND RECOVERY PLANS AND ACTIONS ARE NOT ADDRESSED AS THEY
SHOULD BE IN THESE STATEMENTS. THESE NEED TO BE CARRIED OUT WHILE
ALSO CONTINUING TO ADDRESS ASSESSMENT, REMEDIATION, AND TESTING
CONCERNS. PG

[END OF John Koskinen's statement and
my comments in CAPS regarding his statement]

***************************************************************

The evening panel program that will be held on December 16 at the
Washington Post will be addressing many of the issues raised in this
thread. The topic of the panel is as follows:

"It's Not Over 'Til It's Over and It Could Go On for Years:
Determining Y2K and Embedded Systems Priorities ~
The Need to Continue to Prevent and Minimize Impacts
Now and Into the Future."

I will submitting updated information on the program shortly. In the
meantime you can find information at
http://www.greenspun.com/bboard/q-and-a-fetch-msg.tcl?msg_id=001xgV

There is always a chance that C-SPAN will decide to broadcast the
panel live. Anyone can call or send requests to C-SPAN's viewer
services asking that they broadcast the program (7 PM to 9 PM EST).
Information on how contact C-SPAN is included in the thread just
noted.