Friday, April 3, 2015

Discussion with an Auditor on Lessons Learned

A CIP
auditor emailed me after my recent post on NERC’s Lessons
Learned. His points were all well
taken, and we had a good back and forth discussion. This post attempts to summarize the high
points of that discussion.

RFIs

In the first
post (bullet point number 2 near the top), I said “the NERC Rules of Procedure
don’t permit any modifications to an approved standard, other than through
Requests for Interpretation.” The
auditor pointed out that “a Request for Interpretation CANNOT modify the
language of a Standard. It can only clarify what the language
means. Only a modification to the Standard through the standards
development process can modify a Standard.”
This is correct; I should have said that the RFI is the only
“clarification” of the language that has the same force as the standards
themselves; all other clarifications (including the LLs) don’t have that same
force.

Disagreeing with a Lesson Learned

In the post,
I said “You should definitely consider (Lessons Learned) when you are deciding
how you will interpret a particular piece of wording in CIP v5…However, that
doesn’t mean you should consider the LLs to be the word of God. If you
have a valid reason for believing that the definition of Programmable should be
different from what is stated in the “Programmable” LL, use your version[i]!
But you do need to clearly document,
for future audit, a) that you considered all the available NERC and
Regional Entity documents addressing this issue (especially any Lessons
Learned), and b) the reasons why you chose the particular interpretation you
used.”

The
auditor’s initial response to this in his email was “If you disagree with a LL,
then seek an RFI or submit a SAR.” I
replied to him “As you know, RFIs and SARs work on geologic time, and don’t do
an entity any good for coming into compliance with v5” (by which I meant that submitting
an RFI and getting a response three years from now doesn’t help a lot with
meeting the 4/1/16 compliance date). He
responded, “That may be true, but it is still the only recourse an entity has
to try to change something if they do not like the approved and published
guidance. Otherwise, live with the guidance or risk the consequences.”

I seized on
“approved and published”, and pointed out “I’ll agree that if there is a finalized
LL dealing with the issue in question, the entity would need to have a pretty
good reason for not following it.” But the
bigger issue is that there are only two finalized LLs as of today- one on the far-end
relay question, the other on ‘Impact
Rating of Generation Resource Shared BES Cyber Systems’; neither of these
would be considered fundamental to understanding CIP v5, although they’re both
interesting to certain groups of NERC entities[ii].

But what
about documents other than finalized Lessons Learned? Specifically, a) Lessons Learned that have
been drafted and submitted for comment, but not finalized (currently numbering
around ten); and b) the issues that have been dealt with, or will be, in FAQs[iii]? What is the status of these? Does the entity need to consider these as
something more than just one man’s or woman’s opinion? If so, how much more? Let’s say a finalized LL receives a score of
100 as Something You Need to Pay Attention To – where do these other documents
fall? Does a draft LL get a score of say
50, while a topic addressed in a FAQ is 25?
Maybe those numbers should be 47.2 and 28.9? And isn’t it ridiculous that these would even
be valid questions in the first place?

Most
importantly, what about the hundreds of CIP v5 issues for which there have been
no Lessons Learned or FAQs even proposed?
I listed 20 in this
post last December. I’m sure I could put
together a list of 50-100 now without a lot of work. And I’m sure there have been 500 to 1,000
total issues already identified in various forums and discussions- that number
will keep growing. The entity is
obviously on its own as far as all of these issues are concerned[iv]. You – the compliance person at the entity -
need to ask all the questions you can, read all the documents you can,
etc. But in the end, it’s up to you to
decide these questions (including some quite fundamental ones like how you
identify BES Cyber Systems).

“Programmable Electronic Devices”

This is
probably the most important Lesson Learned currently in the process of being
drafted / approved. A NERC entity,
before it can definitively identify and classify its BES Cyber Systems, must
first identify BES Cyber Assets, and before that Cyber Assets. Since the latter are defined as “programmable
electronic devices”, this means this LL really needs to be in place before
entities can finalize their plans for coming into CIP v5 compliance. So where is the PED Lesson Learned? Nowhere to be seen, although it is promised
Real Soon Now[v]. And when do most NERC entities need to have
it? I’d say at least by January 1, 2015
(not 2016!). Do you think NERC will meet
that date?

This isn’t
anything new, of course. I discussed
the need for this LL last September; it was supposedly in development back
then. Yet what does NERC say about the
delivery date for the finalized PED LL?
According to the auditor, “NERC’s opinion is that even if it is not
finalized for another month, the entities are on notice and they still have
plenty of time to confirm (by which I believe he means ‘conform’ to the LL).” I take this statement as an indication that
whoever from NERC made it has a wonderful sense of humor, although perhaps not
a well-developed sense of irony. The
auditor finds this as frustrating as I do.

This is an
LL that definitely needed to be in place soon after it was proposed last year,
yet it’s still not finalized. But there
is a bigger problem than the PED LL.
It’s all the LLs that haven’t even been proposed yet (that I know of),
yet are sorely needed for entities to finalize their v5 compliance plans (and I
don’t mean finalize their v5 compliance programs; I mean finalize their plans
to reach the point where they can
finalize their v5 programs). Two very
important examples of this are:

Methodology for identifying BES Cyber Systems. I have written many posts on this topic,
including this
recent one. I pointed out in this
recent post that a CIPC group that was developing a LL on a particular
issue that involved BCS identification recently gave up that task, pending
some guidance from NERC on the question of BCS identification in general. So it’s unlikely there can be any real
guidance on any aspect of CIP-002-5.1 R1 until this issue is addressed.

How to resolve the inherent contradiction in the
definition of BES Cyber System without bringing on the need to develop an
RBAM as in CIP v1-3. This was the
subject of my post
last week

Of course,
there are many other LLs that were needed months ago that haven’t even been
proposed. As I said above, the only
choice for entities – rather than wait indefinitely to start their v5
compliance efforts, in the vain hope that all of these issues will be quickly
addressed – is to gather the best advice possible from all sources, then use
their judgment on each of these issues.
And, of course, to document, document, document their entire reasoning
process and conclusions. If you get issued
a PV three years from now for not following a Lesson Learned that came out,
say, at the end of 2015 or early 2016 – when you had to finalize your v5
compliance plans in January of 2015 – I suggest you vigorously fight that PV.
See the last section of this post for what might be an appropriate remedy to
apply should this happen.

Another v5 Compliance Resource

In the
course of debating about the Programmable Electronic Device Lesson Learned, the
auditor pointed out to me a new compliance resource I’d forgotten about:

“If you want
to roll your own, I suggest you dig deeply into the official record. What
is that, you ask? Why, it is the petition to FERC for approval of the CIP
V5 Standards. The filing was in four parts due to its size. The
record of note is Exhibit F, two of the four parts. Exhibit F is the
complete record of the industry comments submitted during the comment and
balloting periods along with the required Standards Drafting Team
response. Surprise, surprise…guess where the definition of PED was
debated and guess where the SDT firmly defined what PED meant. What a
shame they did not make a Glossary definition[vi].
And, yes, you guessed it, FERC read that record and therefore took that
definition into consideration when they approved the Standards. When you
see the next round of guidance on PED, expect to see a definition that follows
the official record. What a shame no one thought about looking there
months ago when the study participants tried to develop a definition independently.
But, you know, hindsight is 20x20. Want to find the record? Go to
the NERC website, click on the filings tab, click on the 2013 archive, and go
to January 31, 2013. Guaranteed to be a titillating thriller of a
read! Might be something entities should read for the very first time!”

I’m
reproducing this paragraph from the auditor’s email as a public service to the
NERC community. The document referred to
is certainly something that everyone needs to know about. Indeed, it’s been on my reading list since it
was submitted in 2013, and it will most likely remain there for some time. While I agree that entities should read this
(not every word of the 2,000 or so pages, of course), I have these concerns:

As I said in this
post last year, any attempt to discern “the intent of the SDT” is suspect
– unless it is confined to the two official records the SDT voted on and
left us. These are the CIP v5
standards themselves (and the new definitions related to them), and the
Guidance and Technical Basis included with each standard. Anything outside of that will be
interesting reading, and may raise some points you may want to consider,
but in the end it is merely someone else’s opinion, even if that “someone
else” happens to be an SDT member who was assigned the task of drafting
this particular section of a document.

If NERC (or one of the regions) wants to use some parts of
the v5 petition as a background for audits, they have the right to do
that. But this needs to be stated
explicitly to the NERC entities involved, and they need to be told which
sections of the 2000-page document are relevant. You can’t just say, “Read the whole
thing. Quiz tomorrow.”

Defenestration

I concluded
the first post with this memorable sentence:

“What should
an entity do with an auditor who suggests that the Lessons Learned (or the
Guidance and Technical Basis in the standards, or the FAQs, or the Bible)
provide some sort of mandatory interpretation of the standards? I
suggest defenestration.”

Since to
defenestrate someone means to throw them out of the window, the auditor –
rightly – took me to task for suggesting physical violence against
auditors. I agree that I should have
made clear that the defenestration I was talking about was a moral one: If an
auditor tells you that the Lessons Learned provide mandatory guidance, you
should make it quite clear to him/her that you do not believe this is the case
(even if the LL in question has been finalized, sealed in blood and deposited
in the National Archives next to the Constitution). And here’s something that may help you: At
the NPCC meeting on CIP v5 that I attended last week, it was stated quite
clearly that the Lessons Learned are merely guidance, not mandatory. Of course, other regions don’t necessarily
agree. Your mileage may vary.

However,
notwithstanding the above, the auditor did tell me that he intends to do all of
his audits on the first floor from now on.

The views and opinions expressed here are my
own and don’t necessarily represent the views or opinions of Honeywell.

[ii]
I also need to say, regarding the far-end relay LL, that either it is badly
written or it is being broadly misinterpreted.
I will have a post out on this soon.

[iii]
The auditor pointed out that “34 FAQ questions and answers were posted for a
45-day industry comment period this afternoon (meaning April Fool’s Day). Comments are due May 15, 2015.” This is good, of course - although I have just noticed a real problem with one of these FAQs that I hope to address in my next post.

[iv]
And whatever you do, don’t even suggest to me that the Small
Group Advisory Sessions are a way to address all of these remaining
issues. They aren’t public and the results
aren’t going to be published, even in a sanitized form. They are certainly “guidance”, but only for
individual entities. If NERC CIP were
really just a set of highly “personalized” requirements, tailored to the unique
situation of each entity, the SGAS would be wonderful. Unfortunately, I can’t find anything in
Section 215 of the Electric Power Act of 2005 (which set up the current
FERC/NERC regulatory structure for the electric power industry) that indicates
Congress intended for FERC and NERC to issue and enforce such personalized
requirements. I think Congress wanted
requirements that would be the same for each entity complying with them, just
as laws and ordinances are meant to apply to everybody. But maybe I’m just too caught up in these old-fashioned
ideas about laws…you know, John Locke and all that.

[v]
Note that these paragraphs were written before the auditor provided me the information
discussed in footnote vi below. I left
it in because I’m an ornery SOB.

[vi]
The auditor did update these comments two days ago. Here is what he said:

“Look for a handful of key guidance documents to be
posted in the next two weeks. Expected to be included is a final,
definitive definition of PED (programmable electronic devices). The
source of the definition can be found in the SDT response to comments submitted
by the industry during the V5 balloting process. You will find it in
Exhibit F, Part 2 of 2, on Page 214 of the PDF file. The response states:

Some commenters asked about “programmable” with
respect to electronic device. The SDT notes that it is an electronic device
which can execute a sequence of instructions loaded to it through software or
firmware, and configuration of an electronic device is included in
“programmable.” Depending on the scheme used, these instructions may include
data, or data that must be processed to execute these instructions.

“A SAR will be required to change the definition.
An RFI will simply point to this memorialization of the SDT consideration of
comments.

“Unlike the recollections of an exhausted SDT member,
this formal response to the comments raised by the industry was memorialized in
the official record and provides the most complete and definitive definition of
PED. It did not say the Chair believes, or some named drafter believes;
the response says the SDT believes… I suspect that a judge overseeing a
hearing or an appeal would find this official record to be helpful and
compelling when compared to an entity’s protestations to the contrary. And
even if entities did not read every word of every comment, they should have at
least read the SDT response. Like a FERC order, where 'the Commission
determines…', there are parts of a record that are much more important.
The SDT response, which is required by the Standards Development Process,
should be mandatory reading for any entity desiring to vote on the
Standards. That is why the process gives entities weeks to cast their
votes.”

My comments on this:

1.This is the first I've heard that NERC will create a SAR (Standards
Authorization Request) to develop a definition of “programmable electronic
device”. However, since it will take a
while for the definition to be finalized (including I believe drafting and
comment periods, a ballot of the membership, as well as NERC BoT and FERC approval),
NERC is giving a pretty broad hint that the definition that results from this
process will follow very closely the passage from their filing of CIP v5 in
2013, quoted above.

2.Unlike the Lessons Learned, etc, this definition
will be the law of the land, just like the requirements themselves and the
other definitions (BCS, Cyber Asset, etc) that were approved with v5. Of course, it won’t really be law until FERC
approves the final product, which could easily be 1-2 years from now. If an entity has already “rolled
their own” definition and built their whole v5 compliance program on that,
I certainly don’t think they have to throw everything away and start over
again, using this new definition.
However, if you are just getting serious now about v5 compliance, I
would definitely recommend using this new definition for PED.

3.I will write a new post on this development –
since I’ve just realized how important this is.
It deserves a lot more than a footnote in a post on another topic.