Monday, June 29, 2015

The News from SPP and WECC: Back to the Far End

June
29 Note: It was recently pointed
out to me that the post below is based on the original version of the “Far-End
Relay” Lesson Learned (posted for comment last September), not the finalized
version that I linked below (which I admit I hadn’t downloaded or read when I
originally wrote the post, since I didn’t realize there had been a big change
in the LL). When I realized my error, I at first thought it would be easy to
just update the post based on the final version. However, when I read the final
version it became immediately clear to me that, while NERC’s conclusion didn’t
change, their rationale did change –and very substantially. Based on the
wording of criterion 2.5, I definitely think the original version had the right
rationale, not the final version.

The conclusion of both versions of the LL is
straightforward: A “far-end” relay, located in a Low impact substation, that is
associated with a 200-499kV line that terminates at a substation that falls
under criterion 2.5, is Low impact – not Medium. The original purpose of this
post was first to state that I agree with the LL (not a big surprise there),
but more importantly to warn against drawing from the LL a conclusion that I
believe would be unwarranted – namely, that all BES Cyber Systems at Low
impact assets are now automatically Low impact.

Fortunately, both versions of the LL support
my warning, even though I don’t agree with the rationale in the final version
of the LL; so the conclusions of my original post don’t need to change. I’m therefore
leaving the original post as it is below, even though it’s discussing the
September draft of the LL, not the final version. But I do discuss the final version of the LL
in the last footnote of this post (BTW, the September version of the LL has
been taken down from the NERC website. If you want to see it, you can email me
at talrich@deloitte.com and I’ll send
it to you).

I must say, this is all quite disappointing
to me. Here I thought I could for once give a clean endorsement to an important
NERC document without any ifs, ands or buts; but now I have to go through an
elaborate dance of saying I agree with the conclusion but not the reasoning of
the final document, and that the document I do agree with has been
officially superseded. I continue to hold out hope that someday I’ll find a
NERC document I can agree with entirely. If that ever happens, you’ll be the
first to know.

This is the
third post in my series on things I learned at the SPP and WECC CIP conferences
the first week of June. I would subtitle the series “What I Did on my Summer
Vacation”, if I could convince you that a week with more than two days of
travel and three days of meetings was a vacation.

I have expressed
my displeasure with NERC for its slowness in coming out with guidance on the
many issues with CIP v5, as well as in many cases for the content of the
guidance it has produced. But there is
at least one guidance document that I consider spot on, in terms of saying exactly
what needed to be said about its subject and not causing any “collateral
damage” by saying more than it should; this document is the Lesson Learned (LL)
on Impact
Rating of Relays, aka the “Far-End Relay” LL (it also happens to be one of
only two LLs that have been finalized).

So why am I
bringing this up? I almost always deal with problems with the rollout of CIP
v5, not with things that aren’t problems. The “problem” with this LL isn’t due
to its content, but to the fact that almost nobody seems to understand what it
means – and this includes people from NERC entities, the regions, and NERC
itself. This lack of understanding can and will likely lead to problems with
implementing and auditing compliance.

The second
paragraph of the LL summarizes the complete argument of the document. It reads:

“As
discussed further below, the language of CIP-002-5 and its support documents
limits the application of the medium impact rating to the BES Cyber Systems
associated with Transmission Facilities operating between 200kV and 499kV at a
single station or substation. The Transmission Facilities must be located ‘at a
single station or substation’ that meets certain connection criteria in order
for the associated BES Cyber Systems to receive a medium impact rating.”

Of course,
this paragraph – indeed the whole LL – refers to criterion 2.5 and only that
criterion. To unpack the content of the paragraph, it says the following:

The subject of the criterion – i.e. what gets classified
as Medium impact – is Transmission Facilities between 200 and 499kV. This
includes lines operated in that voltage range that terminate in the
substation. It does not include
the substation itself; in fact there is technically no such thing in CIP
v5 as a “Medium substation” – all of the criteria that apply to
substations actually classify the Facilities at the substations, not the
substations themselves (of course, in practice it’s almost impossible to avoid
using this language, as I’m about to demonstrate).

Because the “preamble” to Section 2 of Attachment 1 states
that BES Cyber Systems are Medium impact if they are “associated with” the
subject of one of the Medium criteria, this would normally lead one to
conclude that all BCS that are associated with a Medium line at a
substation that has Facilities meeting criterion 2.5 (it would be much
easier to say “a criterion 2.5 substation”, of course) will themselves be
Medium impact. And this would
include “far-end” relays in a transfer-trip scheme, even if these are
located at a substation that is otherwise Low impact (and yes, a
substation can itself be Low impact. In fact, no Facilities are Lows, just
assets are. The wording of CIP-002-5.1 is contradictory on this point, as
on others).

When this implication became widely known, there was a
great hue and cry that this would lead to huge costs for transmission
entities, as they would have to spend lots of money to protect these
Medium BCS at Low substations. However, the Lesson Learned (released last
September) made it clear this won’t happen. To see NERC’s reasoning, just
look at the paragraph quoted above: It points out that in criterion 2.5 the
word “Facilities” is modified[i]
with the words “at a single station or substation”.

This means that, for this
particular criterion[ii],
all lines are excluded from being Medium impact Facilities, since they are
inherently not limited to a single station or substation. Because the line
isn’t a Medium Facility, the far-end relay can’t be considered a Medium
BCS, since it isn’t associated with a Medium Facility. About three months
before this Lesson Learned was released in its first draft, exactly the same
argument had appeared
in my blog, contributed by an Interested Party who has often contributed
to my posts.[iii]

So what’s
the problem? The problem is that many people in the NERC community – I’m
willing to bet it’s the majority, although I haven’t conducted a survey –
believe that what the LL really says is something like “Location does matter”;[iv] in
other words, that all BES Cyber Systems that happen to be located at Low impact
assets are therefore Low impact simply because of that fact. This is absolutely
not the case; the Lesson Learned only applies to BES Cyber Systems (probably
always relays) associated with lines that terminate at a substation that
“meets” criterion 2.5. It doesn’t apply to anything else.

Does this
have a real-world impact? Yes, it does. Here are examples of two systems,
located at a Low asset that might actually be Medium BCS:

Suppose you have a centralized system – located at a Low
impact substation - providing access control for cyber assets at
substations, including some Medium BCS. Would the access control system be
a Medium BCS? I believe it would, since it would presumably be associated
with the Medium Facilities (lines, etc) that the Medium BCS it controls
are associated with (in other words, “guilt by association”).

Or suppose the Automatic Generation Control (AGC) system
for a Medium plant is located at a Low impact plant, substation or control
center. Since it’s associated with a Medium plant (perhaps meeting
criterion 2.3), it will itself be Medium impact.[v]

Note: An Interested Party pointed out that both of the examples I just gave are fairly unlikely to occur in practice. He pointed out that one very real example is an SPS/RAS system that meets criterion 2.9. The different components of the SPS - each a BES Cyber Asset in its own right - could be located at a number of different substations and/or generating plants that are Low impact. However, since the SPS (now officially called RAS, I believe) is an asset (one of the "magic six") that is Medium impact by 2.9, all of its component BCS will be Mediums as well - regardless of whether they're located at a Low or "Medium" impact asset.Here's a note on my note: As I wrote the note, I was trying to figure out the implications of calling SPS an "asset", when it is actually really a system, with components spread out among multiple assets. It would be nice to figure out exactly how SPS/RAS fits into the admittedly shaky "system" of asset identification and classification in CIP-002-5.1. I'll put that on my list of posts to work on. If anybody has any particular thoughts on this matter, let me know.

The views and opinions expressed here are my own and don’t
necessarily represent the views or opinions of Deloitte & Touche LLP.

[i]
And if you don’t remember what “modified” means here, please dig up your
sixth-grade textbook on diagramming sentences.

[ii]
The same phrase appears in Criterion 2.6, where it would most likely also have
the effect of removing Transmission lines from consideration.

[iii]
You may wonder what happens to the “near-end” relay – i.e. the one that resides
in the “Medium” substation and is associated with the 200-499kV line; is that
now also Low impact? This would be true if it were only associated with the line. However, it is also associated (and
more closely, too) with the circuit breaker that can trip that line. Since the
circuit breaker would be a Facility operated at 200-499kV at a substation that
has 3,000 points, then the relay is a Medium BCS. And BTW, if the far-end relay
directly controlled that “near-end” breaker, I would say that relay would then
be a Medium BCS, in spite of being located at a Low substation. Fortunately, I
don’t think this is generally the case with transfer-trip relay schemes.

[iv]
These were Tobias Whitney’s words when he “explained” NERC’s position on this
issue at the June 2014 CIPC meeting, as described in this
post. Those words seem to have taken on a life of their own, even though the
Lesson Learned uses a very different argument – both the draft and final
versions.

[v]
Here’s my footnote on the final version of the Lesson Learned. I must say, this
version is an odd document. It seems to make two slightly different arguments,
both leading to the same conclusion. I
don’t agree with either argument, but as I said in my note at the top of the
post, the good news is that the overall conclusion of the final version of the
Lesson Learned is the correct one. This
is also the conclusion of the September draft of the LL – and I agree with that
document 100%.

The first argument is in the first paragraph of the LL,
which states that relays “located at Transmission stations or substitutions (sic – I’m guessing NERC meant to say
‘substations’ here) described in criterion 2.5” should be Mediums, while those
located at substations that don’t meet 2.5 (or any of the other Medium
criteria) should be Lows. My problem
with this is it seems to completely ignore the fact that criteria 2.3 – 2.8
apply to Facilities, not to assets. I’ll say this for probably the 15th
or 20th time (and the second time in this post): the SDT didn’t put
the word “Facilities” in those criteria just because they wanted to break up
the monotony – they did it because they wanted Facilities to be what those
criteria apply to, not assets. Facilities are lines, breakers, transformers,
etc. Assets (with a little “a”) are the substations, generating plants, etc. So
the substations don’t technically “meet” criterion 2.5 or any other criterion.
This can have consequences for the amount of work the entity has to do to
comply, but I’ve also discussed that issue at length, such as in this
post.

More specifically, reading this first paragraph of the
final version of the LL will lead one to conclude that the entire determinant
of whether a BCS is Medium or Low impact is the substation it’s located at.
This is simply not true. For example, in a criterion 2.5 substation, a relay
associated with a circuit breaker operated at less than 200kV will be Low
impact, not Medium.

The last paragraph seems to bring up a different argument,
although it also leads to the same wrong conclusion as the first paragraph. I
quote that paragraph in full:

“The
Guidelines and Technical Basis (Guidelines) section of the Reliability Standard
also discusses Transmission
Facilities described in Attachment 1 which states: ‘In most cases, the criteria
refer to a group
of Facilities in a given location that supports the reliable operation of the
BES. For example, for Transmission
assets, the substation may be designated as the group of Facilities.’ According
to the Guidelines,
‘The Transmission Facilities at the station or substation must meet both
qualifications [i.e., the connection specifications described above] to be
considered as qualified under criterion 2.5.’”

I
actually agreed with the SDT when they wrote in the Guidance and Technical
Basis that a substation could be considered a “group of Facilities” – that’s
about the best definition of “substation” you could come up with (my usual
definition is “a bunch of expensive equipment with a fence around it”). It’s
hard to say what the above paragraph is saying, but it seems that, instead of
moving from this observation to the conclusion that the near-end relays are
Medium if and only if they’re associated with a Medium Facility, whoever wrote
this LL seems to be falling back on the idea that all the BCS at a “Medium”
substation should be classified as Medium impact, regardless of whether or not
they’re associated with a Medium Facility. By implication, they’re also
implying that all BCS at Low impact assets will be Lows. Neither of these
statements is true.

The
ironic part is that whoever wrote the draft LL from last September seemed to
understand quite clearly that it is the Facility that determines the impact
level in criterion 2.5 (and by implication in criteria 2.3 – 2.8). I’d love to
know why this understanding has been lost to NERC. It’s kind of like if Apple
had suddenly forgotten how to make smart phones.