Tom Alrich's Blog

Friday, February 16, 2018

Five days
ago, I wrote a post calling attention to Lew Folkerth’s December article on
CIP-010 R1 (configuration management) compliance. In that post, I mentioned
that you can now sign up to receive notices of new RF newsletters (where Lew’s
column appears). I signed up and I’m glad I did, because yesterday I got notice
of a new newsletter.
This time, Lew’s article (which as usual goes under the name “The Lighthouse”)
is about CIP-007 R2 patch management mitigation plans. Lew’s articles are
always excellent, but this is still one of his best yet, in my opinion.

I must admit
I hadn’t thought very much about patch management mitigation plans. R2.3 says “Mitigation
plans shall include the Responsible Entity’s planned actions to mitigate the
vulnerabilities addressed by each security patch and a timeframe to complete
these mitigations.” I had always just assumed this was all you need to know
about these plans.

It turns out
I was wrong. Lew does an excellent job of pulling out what really needs to be
in a mitigation plan and how it needs to be handled. None of this is in any way
an add-on to the requirement, by the way. Lew is simply following the logical
implications of what the mitigation plan needs to accomplish, both for security
and compliance. Lew has always been most concerned about what makes the most
sense from a security point of view. I won’t say everything he says in this
article is needed strictly for compliance with the requirement, but doing it
will certainly allow you to tell a good story when you get audited.

If you would like to comment on what you have read here, I
would love to hear from you. Please email me at tom@tomalrich.com. Please keep in mind that
Tom Alrich LLC can help you with NERC CIP issues or challenges like what is
discussed in this post. To discuss this, you can email me at the same address
or call me at 312-515-8996.

Thursday, February 15, 2018

This is the
second in a series of posts of how vendors can “comply” with CIP-013. Of
course, vendors don’t have to comply at all, and they don’t have to help their
power industry clients comply either. On the other hand, if they wish to remain
a vendor to the power industry, a vendor is going to have to do everything they can to help their customers comply. It
is inevitable that the vendor’s larger clients will have to comply with
CIP-013, and they will need help in doing so.

I do want to
emphasize that, if you’re part of a NERC entity who has to comply with CIP-013,
you shouldn’t stop reading this post (or the subsequent ones in this series),
on the idea that it doesn’t really have anything to do with you. This has
everything to do with you, since – as I emphasized in the first
post in the series – the only way I see a CIP-013 compliance program
succeeding is if the NERC entity at least tries to partner with their vendors
for compliance. You need to understand their position as much as they need to
understand yours.[i]

In the first
post in this series, I suggested that power industry vendors would be well
advised to reach out to their customers before
the customers reach out to them. I made this suggestion because I think it’s
very likely that if the customer reaches out to the vendor first, it will probably
be the lawyers and/or purchasing people who do this. This is because most of
the discussion so far on CIP-013 compliance (with this blog hopefully being an
exception!) has revolved around the question of contract language. But contract
language is just one of the ways in which a utility can fulfill its obligations
under CIP-013, and it’s probably the bluntest instrument available to the NERC
entity to fulfill those obligations. I think that any NERC entity that focuses
on contract language first, before even looking at all the options available to
it to comply with R1.2, has already done both itself and its vendors a big
disservice.

So the
vendor should absolutely try to reach out first to their customers. How can
they do that? There are of course lots of media available for this: notice on
web site, email, USPS, phone call, webinar, onsite meeting, etc. While these
are all good, I always favor the more personal ones above all. A webinar might
be the ideal first step, since it’s delivered by a person(s) but it still has a
structured content. Then the vendor could follow up with each customers by
phone or in-person meeting to discuss next steps.

But before
you can do a webinar, Mr./Ms. Vendor, you need to know what you’re going to
say! And that means you need to have figured out your CIP-013 strategy[ii] – so
that’s really the first step. How can you figure out your strategy?

This is of
course rough and preliminary, but it seems to me that a vendor’s strategy
should be to try to make the CIP-013 compliance process as easy as possible for
the customer, period. I can assure you that your customers will be feeling
every bit as uncertain and fearful about CIP-013 as you do. If you can convince
them that you know the right way to cooperate for compliance, and you follow
through on what you say you’ll do, what could possibly go wrong – except,
perhaps, if you don’t in fact know the right way to cooperate and both of you
end up at some sort of regulatory dead end?

How do you
stay away from dead ends? By having a good methodology for designing your
CIP-013 strategy. And what should you do first? Well, it seems to me that, if
you’re going to design a strategy to help your customers do something, you need
to figure out what they have to do. There’s a quick answer to that: They have
to comply with CIP-013. Now that we have that out of the way, how do they comply with CIP-013?

As I
discussed (at length) in this
post, CIP-013 requires the NERC entity to develop and implement a supply
chain cyber security risk management plan. Specifically, the plan has to
address the three areas of risk listed in R1.1: (a) procuring vendor equipment
and software; (b) installing vendor equipment and software; and (c) transitions
from one vendor(s) to another vendor(s).

I suggest
you start your CIP-013 strategy effort by brainstorming on how you can help
your customers address all three of these areas of risk. To be honest, you
might decide you can help customers in some ways that are either too expensive
to be practical or not likely to yield a lot of benefit compared to the effort
required; you can drop these ideas from your strategy.

For
procuring vendor equipment and software, the NERC entity needs to address risk
in six specific areas; these are listed in R1.2. The reason they are
specifically listed in CIP-013 is because FERC specifically required – in Order
829 - that these areas all be addressed in the new standard. The NERC
entity needs to do a lot more than simply address these six areas of risk, of
course, but because they’re specifically called out you can be sure the
auditors will all make sure they’ve been properly addressed in the entity’s
plan, probably before they even get around to looking at anything else.

Let’s choose
one of the areas in R1.2 as an example:

R1.2.4.
Disclosure by vendors of known vulnerabilities related to the products or
services provided to the Responsible Entity;

How can you
help your customers address this area of risk? This is of course a particularly
difficult one for software vendors, since what might seem like the obvious way
to help – disclosing to your customers all of the vulnerabilities in your
software that you know about – would be the height of irresponsibility. If a
vulnerability hasn’t been patched yet, the last thing you want to do is
broadcast the existence of that vulnerability to all of your customers.

On the other
hand, you might decide that you do need to provide information on all
vulnerabilities (not just the ones that have already been patched, which of
course can be advertised to the whole
world) to at least your largest and most trusted customers. You need to decide
internally in what cases you will do this, what safeguards you will require on
the customer end, what alternatives to full disclosure you might suggest to customers
for whom you don’t want to take the “Full Monty” approach, etc.

I hope you
understand what I’m getting at here. For each of the six items in R1.2, you
need to figure out a complete strategy for dealing with all of your customers
(and as you can see, you will probably want to deal with particular types of
customers in different ways, although the way you break down your customers is
likely to vary according to the item in question). This will be an important
component of your strategy, Mr./Ms. Vendor.

But this isn’t
the only component of your strategy. My next post in this series will go over
what else needs to be in your strategy.

If
you are with a vendor to the electric power industry, and your company is
trying to figure out what you will have to do to comply with CIP-013, Tom
Alrich LLC would be pleased to offer you a free one-day (2-6 hours) workshop to
review a) what CIP-013 requires, b) what you are likely to get asked to do by
your clients, and c) what they should be asking you to do (since b and c won’t
usually be the same thing). There will be no charge for my time, but I will
require you to pay travel expenses at cost. Please email or call me if you
would like to discuss this.

If you would like to comment on what you have read here, I
would love to hear from you. Please email me at tom@tomalrich.com. Please keep in mind that
Tom Alrich LLC can help you with NERC CIP issues or challenges like what is
discussed in this post. To discuss this, you can email me at the same address
or call me at 312-515-8996.

[i]
And of course, I’m also doing a series of posts on how NERC entities can comply
with CIP-013. Here
is the most recent post in that series. Vendors should read this series of
posts as well!

[ii]
And by the way, NERC entities also need a CIP-013 compliance strategy, which I
am gradually laying out in my other series of posts. A lot of elements in both
strategies should be the same – just told from different points of view. But a
NERC entity’s strategy will inherently include a lot of elements that have
nothing to do with vendors, since vendor risk is just one of three areas of
supply chain risk that need to be addressed in the supply chain cyber security
risk management plan. See this
post for a short summary of those three areas, and see this
post for a long, painful discussion of that topic – but which ultimately might
be more enlightening.

Wednesday, February 14, 2018

This might
seem like an odd topic for a post on Valentine’s Day, but I’m actually
referring to the anniversary of my blog. A friend asked yesterday how long I’d
been writing the blog, and I at first was going to say three years - because
for some reason I seem to have latched onto that number a couple years ago, and
never bothered to do the advanced math since then.[i] Then I
realized it’s been five years. In fact, just over two weeks ago I missed the fifth
anniversary of my first
post, written in a hotel room in San Diego on January 28, 2013.[ii]

So anyway, I’m
proud that this blog has lasted five years, and I hope it lasts another five! And
for those of you who’ve stuck with me for some or even all of the five years,
thanks!

If you would like to comment on what you have read here, I
would love to hear from you. Please email me at tom@tomalrich.com. Please keep in mind that
Tom Alrich LLC can help you with NERC CIP issues or challenges like what is
discussed in this post. To discuss this, you can email me at the same address
or call me at 312-515-8996.

Any opinions expressed in this blog post are quite
definitely those of my employer, Tom Alrich LLC! If you disagree with what I’ve
said, I suggest you take it up with them.

[i]
This may also explain why for many years I’ve had the idea that I’m 42. I
finally realized that had to be wrong when my son turned 30 last year – and I
was sure he hadn’t been born while I was in junior high!

[ii]
I had previously written posts for a Honeywell blog that no longer exists, and
before that I’d written some “open letters” that Honeywell distributed. The
open letters started in 2010, when I attended my first CIP drafting team
meeting and found it so interesting – and consequential – that I wrote my first
open letter about it.

The topic of my first post wasn't exactly timeless. I was trying to convince people that they had to take CIP version 4 seriously, since after all FERC had approved that version in April of 2012, and the compliance due date was April 1, 2014. I - and many others - felt that CIP version 5 was still a pipe dream that might never be approved by FERC. So it was much better to go with the bird in the hand (v4) rather than the two in the bush (v5). This campaign of mine culminated in what I call my "Dewey Beats Truman" post, when I confidently predicted that v4 would come into effect in 2014, with v5 following a few years later. Of course, just two months later FERC issued their NOPR saying they intended to approve v5, and v4 would go to sleep with the fishes.

Sunday, February 11, 2018

I apologize,
but it seems I’ve fallen behind my minimum quarterly requirement of posts that
quote from Lew Folkerth of RF. I just discovered Lew wrote a great article on
configuration baselines and CIP-010 R1 for the RF Newsletter
dated November/December 2017. You can find it by clicking on The Lighthouse in
the table of contents on the left side of the page. I was also pleased to note
that RF will now send out emails when new newsletters come out (which is
bi-monthly), so neither you nor I will miss any future articles from Lew.

The article
speaks for itself, but here are the points I found most interesting[i]:

Installed software and firmware listed in CIP-010 R1
should match software and firmware listed in CIP-007 R2 (Patch
Management). Auditors check for this now, so you should definitely make
sure they match on a regular basis, and even sync the two lists up if possible
(page 16, last column).

A good tip for simplifying the job of CIP-007 R1 (Ports
and Services) documentation by leveraging information from the baseline (p.
17, first column).

Benefits of a good baseline for incident response (p. 17,
first column).

Lew’s recommended list of software and firmware to include
in the baseline (p. 17, third column).

Lew recommends that firewall rules be under change
management, whether or not they’re included in the baseline for the
firewall.

The box about scripts on page 18 is worth the price of
admission by itself! And that certainly doesn’t mean it’s worthless, even
though admission to the article is free.

I recommend you all read the article, as well
as subscribe to the newsletter.

If you would like to comment on what you have read here, I
would love to hear from you. Please email me at tom@tomalrich.com. Please keep in mind that
Tom Alrich LLC can help you with NERC CIP issues or challenges like what is
discussed in this post. To discuss this, you can email me at the same address
or call me at 312-515-8996.

Any opinions expressed in this blog post are quite
definitely those of my employer, Tom Alrich LLC! If you disagree with what I’ve
said, I suggest you take the matter up with them.

[i]
A few of these aren’t new – in fact, I’ve written about them in previous posts)
– but they’re worth repeating.

Thursday, February 8, 2018

The title of
this post should give you pause. Vendors don’t have to comply with CIP-013;
only NERC entities do. But this is a distinction without a difference. It has
always been the intention of both FERC (in ordering NERC to develop a supply
chain security standard) and the NERC standard drafting team (in developing the
standard) to require vendors to be involved with compliance – although there
will be no penalty to the NERC entity if they try to get a vendor to cooperate
in compliance and are unsuccessful.

So it’s very
legitimate to ask how a vendor can comply with CIP-013. I have had discussions
with several vendors (and many more NERC entities) about this, and here is what
we have come up with so far. I do want to point out that this is quite
embryonic.

The most
important consideration for a vendor is that it will be a big mistake if you
look at CIP-013 compliance as some sort of adversary exercise, although you
should be forgiven if you have thought of it that way so far. After all, a lot
of the guidance so far (and that’s just one 13-page document) focuses on
contract language.

My idea of a
contract language “negotiation” is one in which the NERC entity’s lawyers and
the vendor’s lawyers are sitting on opposite sides of an 8-foot-high brick
wall. The entity’s lawyers throw some contract language to the other side and
snarl “Here, put this in our contract.” The vendor’s lawyers look it over and
throw something else back, saying (perhaps snarling less) “We can accept some
parts without change and some with changes, but we can’t accept these other
parts at all.” The entity’s lawyers look this over, and reply with their own
counter-proposal. This goes back and forth, with the length of time determined
by whether or not the lawyers are on salary (in which case they can do this for
years) or they are paid by the hour (in which case the company will have to
call an end to the fun and tell them to wrap up the “negotiation”).

Of course,
this is perhaps an overly bleak view of the contract negotiation process, but I
think it illustrates what I’m saying: Both the vendor and the entity should do
everything they can to keep their relationship from degenerating to this point.
In fact, IMHO if they first resort to contract language in their CIP-013
compliance process, both sides have
already lost the game. Contract language is a nice thing to have, but it should
never be the first concern.

So what’s
the best way for a NERC entity and a vendor to approach the CIP-013 compliance
process? I see two possible ways to get this going:

The entity reaches out to the vendor and says “Let’s have
a discussion on how we (note the
plural pronoun!) can address CIP-013 compliance.”

The vendor reaches out to all of their power industry
customers (at least all who have any High or Medium impact assets where
their software or hardware product is used), perhaps through an e-mailing
or even a – gasp! – USPS mailing, and actually suggests how both sides could work together to achieve the
goal of CIP-013 compliance.

So here’s a
little quiz: Which of these two actions is more likely to result in a fruitful
partnership to achieve CIP-013 compliance? I think they are both good options,
but I would strongly recommend that the vendor be the first to reach out.
Because if the entity reaches out first, I can almost bet that most of them
will do that in the form of a demand for particular contract language. This isn’t
because they’re sociopaths, but because NERC compliance up until now has
primarily been seen as a legal exercise, whether for NERC CIP or the other NERC
standards.

If the
vendor reaches out first, and if they don’t immediately focus on contract
language (if they do, they’ve obviously decided that it’s important that
CIP-013 compliance be made as painful as possible, probably for reasons of job
security of the people writing the document), then they can set the terms of
engagement. The vendor can suggest actions that favor its strengths – i.e.
developing, supporting and securing great hardware and/or software products –
and don’t favor the utility’s strength (which is the fact that they are, after
all, the buyer, and that money is flowing from them to the vendor, not vice
versa. Plus the vendor doesn’t have a monopoly, at least in any electric power
product market that I know of).

What should
the vendor suggest to their power industry clients? I’ll leave that for the
next post in this series.

If
you are with a vendor to the electric power industry, and your company is
trying to figure out what you will have to do to comply with CIP-013, Tom
Alrich LLC would be pleased to offer you a free one-day (2-6 hours) workshop to
review a) what CIP-013 requires, b) what you are likely to get asked to do by
your clients, and c) what they should be asking you to do (since b and c
won’t usually be the same thing). There will be no charge for my time, but I
will require you to pay travel expenses at cost. Please email or call me if you
would like to discuss this.

If you would like to comment on what you have read here, I
would love to hear from you. Please email me at tom@tomalrich.com. Please keep in mind that
Tom Alrich LLC can help you with NERC CIP issues or challenges like what is
discussed in this post. To discuss this, you can email me at the same address
or call me at 312-515-8996.

Any opinions expressed in this blog post are quite
definitely those of my employer, Tom Alrich LLC! If you disagree with what I’ve
said, I suggest you take it up with them.

Tuesday, February 6, 2018

I was
honored to be asked by Indegy – a company you should really learn about,
especially if you own or operate power plants – to join nine other industrial
cyber security “influencers” in giving our ideas on what you, i.e. a staff
member in an organization operating ICS, can expect in 2018. While I can’t tell
you whether your taxes will be lower or you’ll receive a bigger bonus this
year, I can talk about what I think is an important development – which anyone
who follows this blog might have already heard about once or twice J.
You can read the post, on Indegy’s blog, here.

If you would like to comment on what you have read here, I
would love to hear from you. Please email me at tom@tomalrich.com. Please keep in mind that
Tom Alrich LLC can help you with NERC CIP issues or challenges like what is
discussed in this post. To discuss this, you can email me at the same address
or call me at 312-515-8996.

Friday, February 2, 2018

If you’ve
read the first two posts in this series (here
and here),
you realize that for these posts I’ve tried to forget everything I know or have
written about CIP-013, and just work through the words of the requirements (and
the Purpose statement at the beginning of the standard). My goal has been to
mine what is actually written in the requirements and get it all out on the
table. The reason I’m doing this is because nothing is binding on the auditors
(and therefore on the NERC entity) except what is in the requirements. So it’s
important to put in a lot of effort to pull apart the requirements to find
everything that’s possible to find in them – before we start going beyond what
the wording says and start to think about the best way to actually comply with
the requirements.

In our last
episode, we wallowed in R1 and found – to my genuine surprise – that there is
something important missing from R1.1: While this requirement mandates that the
NERC entity “identify and assess” cyber security risks in the supply chain, it
doesn’t require them to do anything about them! Of course, this is clearly just
a mistake, and I highly recommended that everyone assume the word “mitigate” is
also in there, whether or not this problem ever gets fixed in a formal manner.

Now we’re
ready to go on to R2, which on the surface seems quite simple:

Is this
really that simple? Yes and no. It isn’t simple because of what I just pointed
out about R1.1: Your supply chain cyber security plan must include mitigation
of the risks you’ve identified and assessed, even though that step isn’t included
in the requirement part; so you need to mitigate risks when you implement the
plan as well. But it is simple because it doesn’t mandate anything about how
the plan needs to be implemented.

It seems you’re
on your own in this requirement. You have to implement the plan, but you might
take one year or 50 years. You might complete one part before starting on the
next one, or you might address all parts simultaneously and try to complete
them all at the same time. You might prototype your plan in one part of the
company before moving it to others, etc. It seems the details are all up to
you.

Indeed, this
idea is reinforced in spades by the Implementation Guidance document put out by
the CIP-013 drafting team last year. The document says literally nothing about
what must be done to implement the plan, beyond repeating the substance of the
note that appears below R2, emphasizing that the entity isn’t required to
renegotiate or abrogate existing vendor contracts. It seems the drafting team
couldn’t think of anything important to say about implementation.

However, let’s
be realistic: The details of implementing a NERC CIP standard are never up to
you. Last fall, I looked at the industry’s early experience with audits of
CIP-014; like CIP-013, this standard requires the entity to develop a plan (in
this case, a physical security plan for key substations) and then implement it.
In two posts (here
and here),
I told several stories that point to the same conclusion: Even though R2 lacks
any details about what constitutes a good vs. a bad implementation, the auditors
are going to have their own ideas, and they won’t hesitate to make these known
to the entity if they think they haven’t done a good job of implementation[i]. They
hopefully won’t actually issue a Potential Non-Compliance (PNC) finding, as
they did to one of the entities discussed in the second post. But they will very
likely identify an Area of Concern and require the entity to fix the problem
identified.

So what are the
auditors going to look to when they decide whether or not an entity has
properly implemented their supply chain cyber security risk management plan
from CIP-013 R1? Darned if I know, but there is one thing of which I’m sure: I’m
sure that auditors will look at the actual timeline for completing the
implementation, and how that compares with what is in the plan.

Say your
plan lays out a two-year implementation schedule, and your next audit is
scheduled for about 1 ½ years after you start the implementation. If the
auditors show up on that date and decide there’s no way your implementation can
be completed by the two-year mark, they will very likely issue an Area of
Concern, meaning you will need to develop a new plan with a realistic implementation
date. And you’d better make that new date, or you’ll be in trouble three years
later at your next audit.

So what does
this mean for your plan? Should you always sandbag the finish date in the plan,
making it at least a year or two after the date you think you’ll really finish?
I don’t really think this is a great idea. I think you should include in your
plan what you honestly believe is the date you will finish your implementation.
However, I also think you should keep very close tabs on the implementation.
Whenever it begins to look like you will not be able to meet your original
finish date, you should immediately revise the plan to reflect the new date (of
course, you should have change control to document that you made this change. You
certainly don’t want to be in the position of needing to convince the auditor that
the revised plan was actually the original one!).

Of course,
if the auditor doesn’t agree with your excuse for why the implementation date
had to be moved back (e.g. “My dog ate our supply chain cyber security risk
management plan and we didn’t have any backup. This forced a six-month
implementation delay as we re-drafted the plan from scratch.”), they may
disagree with you. But IMHO there is no way they can issue you a PNC because of
this, since R2 is completely silent on what characterizes a good vs. a bad
implementation.[ii]

What’s the
moral of this story? Even though R2 looks like it’s wide open to whatever
interpretation you might want to give it, this doesn’t mean you should simply
do whatever you want when you implement your supply chain cyber security risk
management plan. The auditors will
always have their idea of how a plan should be implemented. It is almost
certain that they will make their ideas clear to you in your audit, even if –
hopefully – they don’t resort to a PNC to do that.

But what if
it didn’t need to be this way? What if there were a way you could get your auditors
(or at least your Regional Entity) to weigh in on whether and how your plan
should be changed, before you started
to implement it? That is the question I discussed in this
recent post. There needs to be some way a NERC entity can get their CIP-013 R1
plan reviewed by their Region before they embark on implementing it. Moreover,
there needs to be some way the Region can check in on the implementation of
that plan as it is in progress, to make sure the entity hasn’t made some big
mistake that might force it to re-do some or all of what it has already done.

The post
described, with the assistance of an auditor, the Entity Development teams that
are already implemented in one NERC Region and being implemented in another.
The members of these teams aren’t auditors, but they help entities comply with the
NERC CIP standards (and possibly other NERC standards). It seems to me that
these teams would be the ideal organization to review an entity’s CIP-013 R1
plan as well as its implementation, before the entity has invested a sizable
sum in implementation of what could turn out to be a flawed plan

I believe
that some mechanism like this needs to be in place, in order for CIP-013 (as
well as other plan-based standards like CIP-014 and the upcoming CIP-012, along
with plan-based requirements like CIP-003 R2 and CIP-010 R4) to be successful.
I hope you agree with me on this (and if not, that you’ll let me know). I hope
NERC does, too!

If you would like to comment on what you have read here, I
would love to hear from you. Please email me at tom@tomalrich.com. Please keep in mind that
Tom Alrich LLC can help you with NERC CIP issues or challenges like what is discussed
in this post. You can email me at the same address or call me at 312-515-8996.

[i]
To be honest, the auditor issues in the second of the two posts cited (i.e.
Part 3 of the series on lessons from CIP-014) had to do mostly with the plan
itself, not with its implementation. But that is just because of timing: The
two audits discussed in that post both occurred before the entity had even been
able to start to implement their plan. Of course, this was very beneficial, since
– had they implemented the plan, which was later determined to be insufficient,
they might have been forced to re-do some or all of their implementation, using
a new plan that was deemed sufficient.

[ii]
They might conceivably issue you a PNC for violating CIP-011 R1.2, since the
plan presumably included BES Cyber System Information and you obviously didn’t
have a very good plan for protecting BCSI from your dog.