Faceți căutări pe acest blog

VULNERABILITY METRICS AND KPI

KPI definition

A key performance indicator(KPI) is a measure of performance,
commonly used to help an organization defineand evaluate how successful it is,
typically in terms of making progress towards its long-term organizational goals.

–KPIs provide business-level context to
security-generated data

–KPIs answer the “so what?” question

–Each additional KPI indicates a step
forward in program maturity

–None of these KPIs draw strictly from security data

COBIT

Control Objectives
for Information and Related Technology (COBIT) is a
framework created by ISACA for information technology (IT) management and IT
governance. It is a supporting toolset that allows managers to bridge the gap
between control requirements, technical issues and business risks.

COBIT was first released in 1996; the current version, COBIT 5, was
published in 2012. Its mission is “to research, develop, publish and promote an
authoritative, up-to-date, international set of generally accepted information
technology control objectives for day-to-day use by business managers, IT
professionals and assurance professionals.”.

COBIT, initially an acronym for 'Control objectives for information and
related technology', defines a set of generic processes to manage IT. Each
process is defined together with process inputs and outputs, key process
activities, process objectives, performance measures and an elementary maturity
model. The framework supports governance of IT by defining and aligning
business goals with IT goals and IT processes.

The framework provides good practices across a domain and process
framework.

The business orientation of COBIT consists of linking business goals to
IT goals, providing metrics and maturity models to measure their achievement,
and identifying the associated responsibilities of business and IT process
owners

COBIT 5 is the only business framework for the governance and management
of enterprise IT. This evolutionary version incorporates the latest thinking in
enterprise governance and management techniques, and provides globally accepted
principles, practices, analytical tools and models to help increase the trust
in, and value from, information systems. COBIT 5 builds and expands on COBIT
4.1 by integrating other major frameworks, standards and resources, including
ISACA’s Val IT and Risk IT, Information Technology Infrastructure Library
(ITIL®) and related standards from the International Organization for
Standardization (ISO).

Benefits

COBIT 5 helps enterprises of all sizes:

-Maintain high-quality information to support business decisions

-Achieve strategic goals and realize business benefits through the
effective and innovative use of IT

The relationship between governance and management in COBIT is presented
in the next figure.

Figure 2. Relationship between governance and management in COBIT 5(Courtesy to COBIT)

The governance and management key areas are presented below.

Figure 3. Governance and management key areas in COBIT 5(

(Courtesy to COBIT)

The relationship between KPI and COBIT is shown in the next figure.

Figure 4. Relationship between KPI and COBIT

(Courtesy to COBIT)

The principles that are founding COBIT are presented in the next figure.

Figure 5. COBIT principles

(Courtesy to COBIT)

How to choose KPI in relationship with vulnerability

KPI’s must be aligned at least with a business goal:

· KPI’s must be in accordance with risk or targets that are recognised by the various stakeholders[1].
When the (business) risk or objective is not lucid a KPI will not come alive.

KPI’s are measured by measures. It
is desirable to not define the KPI too specifically so as to avoid the KPI
influencing the selection of measures still to be implemented.

Some questions could better define KPI.

· Are the KPI’s meant for internal use or for external communication to
customers, partners, shareholders, affiliated organisations, certifying bodies,
etc.?

Must they be connected to other standards in the sector so that KPI’s are somewhat
comparable between organizations or is an organization free to choose their own
definition?

Even when using the same framework (for example Cobit) the comparability
of KPI’s will never be optimal. Organisations will always, in practice, make a
translation to their own situation.

· How generic or how specific must a KPI’s definition be? Are they
applicable to the whole company or just for a department?

In a centrally run organisation a KPI can be much more generically
defined across n different process than in a de-centrally run organization.
After all in a de-centrally organised organisation similar processes will
differ more and therefore so will the measurability and steerage possibilities.
And KPI’s must fit optimally with a process in order for it to be effective.

· Are KPI’s stable during the life cycle of the subject to be
measurement?

During the life cycle of an application, for example, the need for
measurements shifts.KPI’s that play a role in the development phase of
applications differ from those in the operational phase. Also the target group
being reported on can shift during the life cycle.Measurements must be trustworthy
and meaningful

· What maturity level does an organisation or a process have and
which KPI’s are useful for this maturity level?

Controlled measurement of certain KPI’s demands reproducibility
and a tight framework in the execution
of administration processes. KPI’ s meant for tuning of an administration
process that is not (yet) there, have little additional value. The accumulation
of detailed measurements and trend information concerning intrusions, for
example, is of little use when an organisation does not have

its basic incident management process in order.· The measurability of the desired information is important.

The filling in of a security paragraph in a project
phase document is simple to measure. Whether the quality of the paragraph is
sufficient, is already more difficult to determine and without the involvement
of security experts will probably not lead to accurate and objective
information.

· In all scenarios the integrity of measurements is of great
importance. The costs of the measuring
must be in balance with the eventual objective. Often tooling is required to
keep measurements cost-effective during a longer period of time, to guarantee
integrity and to keep reports consistent. With manual aggregation of information,
manipulation is possible and reliability cannot always be guaranteed.

-KPI’s on different levels must be correlated and provide a consistent
view. Multiple

KPI on a lower level can be translated to one KPI on a higher level.
Because these

KPI’s are usually geared towards another target group with a higher
aggregation level,

word usage in the explanation must often also be adjusted. The word usage must fit in

with the actual world and
interests of the target group.KPI must be easy to understand. Higher management may for example be more accountable for the
percentage of application development projects where a risk analysis was
carried out in due time, than for the completeness and depth

of an analysis carried out for a specific development project.

· Aside from standard reports there must be a mechanism for exception
reports.Exceptions are important to keep people awake. In the case of exception
reports it is often effective to give people ‘the naked truth’. Allow
the shop floor to speak and do not try to make the reality prettier than it
actually is. If for example, the management of a department does not tackle
events that structurally occur and remains vague about it in his reports, it
can be worthwhile to magnify an incident with

more than average impact. Diligence and care for disclosure of sensitive
information remains an important factor.

· The chance that Security Management KPI’s are effectively used is
greatest when applied to processes where people are already used to reporting
other quality aspects by means of KPI’s.

This is reason to define Security Management KPI’s based on a process
approach and to assure that KPI’s are built into processes where security is
relevant, such as incident management and change management.

KPI design. KPI developed for vulnerability analysis

When an organisation wants to begin to define KPI’s it can generally
choose two approaches,the top-down approach, which
requires management commitment, and the bottom-up

approach
whereby initial reports based on measurements at hand are the stimulant for
more

and sharper reports.

Top-down

The basis and starting point in the model is obtaining a strong top-level
management

commitment. Subsequent policies and measurements must be defined.

The top-down
approach will work especially when management is experiencing great pressure,
like for complying with SOX regulation on time.This is only attainable in the
short run with

sufficient
management commitment.This approach will, in nearly all cases, be based on
checklists from existing models like Cobit.

Bottom-up

An alternative
approach is to ‘just start anywhere’ with presentation of easily obtainable

measuring results
from operational processes. In this way management is made aware of the

interesting
information reported from processes which can be used for steerage. Initially
the

information
offered will not suitably comply with the information requirement of the
process

owner, but the
imagination of the process owner will be stimulated and he will be in a better

position to define
a modified KPI which is more in line with the requirement.

By means of an
iterative process, KPI’s can be constantly improved. Important in this

bottom-up approach
is that expectations must be well managed to avoid the threat of a

process owner
quitting prematurely and losing his confidence.

An example of a bottom up approach is given below.

No

Actions

1

Determine
security maturity e.g. by examining Stacey’s security model

2

Keep
registration of:

· Incidents (reactive)

· Identified risks (proactive)

Classify these e.g. by sort, object, department.

Also think outside the borders

3

Personalise
reports to target groups

For example: Legal matters, HRM, financial affairs

Aim is to increase awareness

4

Explaining
reports per part area

- Point out consequences per responsibility area

Refer to damage table, in which damage categories are defined

5

Expand
this:

· Formalise KPI’s: Where is the threshold?

· Ascertain trends

· Provide steerage by coupling trends to advised measures

We have defined the following KPIs for vulnerability:

Vulnerability level (Vl)- Vulnerability
level expresses the overall vulnerability for the whole company or for the
section analysed. Vulnerability level could be measured on a scale from 1- no
significant vulnerability to 5- extreme vulnerability. Vulnerability level is a lagging KPI.

Vl=(VbO+VbOP+VbM+VbT+VbE)/5

The oposite of Vl
would express the level of preparedness
against the action of vulnerabilities.
The KPI corresponding is:

Preparedness level (Pl)= 1/VlPreparedness
level is a leading KPI.

Vulnerability level is given by other 5 lagging Kpi as seen in the
figure.

Figure 6. Components of the Vulnerability level KPI

All the components are lagging KPI.

Vulnerability by Outsider (VbO)-
gives the enterprise vulnerability to the attack by outsiders. Such attacks
could be direct (physicall) or could be virtual, through the network of the
enterprise, towards the command and control functions, etc. The more the
enterprise is interconnected in a large technical and societal network ,the
more it is exposed to attacks by the outsiders. There are two main kind of outsider attackers:

-with financial
motivation- are looking for financial awards as a result of the attack;

-with other
motivation such as revenge, curiosity, etc.

Financial
motivated attackers could be more easy identified and their attack prevented. The
most valuable assets of the enterprise must be guarded and proteged . Redundant
solutions should be ready in the case of failure of critical significant
systems.

Vulnerability by Operator (VbOP)-describes
the vulnerabilities introduced in the system by the employees. The operator
could be trained insufficiently or incorrectly, he could make mistakes because
its physical and psychical state, could
be overstressed or affected by events that are not connected with the work. On
the other side , the operator could make intentionally mistakes, for various
reasons, from financial ones (for example he introduces a speed of work at
machine X greater than the prescribed parameter in order to gain more money)
till the ones connected with his relationship with the management.

Figure shows the
main elements that are leading to vulnerability by operator.

Figure 7. Vulnerability by operator

Vulnerability by Machine (VbM) is
given by the defects and failures that could appear during the work process at
the machines and instalations that are used in this process.It could be written
that:

VbM=f(Complexity, Protection, Work regime, Capability of repair)

Vulnerability by
machine[2] is
a function of complexity of the machine (the greater complexity leads to a
greater vulnerability to fail), existing protection systems, the work regime in
which the machine performs (better performances are requiring a more intensive
work regime which could overpass the prescribed thresholds) and also of the
training of the operator and the capability of repairing the machine in work.

Vulnerability by task (VbT) is specific to
the task being developed, transmitted and processed by the operator.Figure
shows the components for this type of vulnerability

Figure 8.Vulnerability by task

Vulnerability by Environment(VbE) is
given by the environment influnece on the work and on the global vulnerability.
If the work is done outside or in closed spaces or in noxes that are not
properly captured, etc. it could lead to loss or even incidents and work
accidents. For example a bad insulated ceiling could lead to a waterfall over
the electric instalation. Work in open spaces could lead to health problems,
etc.

A number of 5 specific KPI were defined as follows, based on the
Capability Maturity Model (CMM):

DRW–Defect
Remediation Window=The length of time from when a vetted (web) application
security defect is identified until it is verified closed-Maturity level:2

Facts
regarding DRW:

–#1 most critical KPI

–DRW will be potentially very large at
first

–Critical to shrink this metric as
quickly as possible

–Can be used to target education where
needed

–Important to note type of defect remediated (complex defects
take longer to fix)

RDR–Rate
of Defect Recurrence=The rate, over time, at which previously closed (web)
application security defects are re-introduced into a given application,
organization, or other logical unit-Maturity level:3.

Requirements:

–Advanced defect tracking system

–Advanced web application security
testing capabilities

–Capabilities to identify similar or likedefects across an
application or logical trackable unit

Facts
regarding RDR:

–Reappearing defects measure internal
development confusion

–Recurring defects should prompt a
systemic investigation into root-cause

Comentarii

Postări populare de pe acest blog

Acknowledgements:
The author wants to thank XpertRule Software LTD and mr. Tim Sell for being
able to try Decision Author- the main software in which this prototype shall be
built.
GENERAL ASPECTS
Safety domain of research is by excellence a domain based on
expertise. Textbooks and theoretical knowledge are good but the safety expert
which inspects three times a day a certain part of an enterprise is the
ultimate safety dealer here.
A lot of expertise is transformed into lessons learned- that
are used for training and improvement of existing safety attitudes. On the
other part, this expertise could be also valued in order to build optimal and
effective safety assessment systems.
An expert system is software that emulates the
decision-making ability of a human expert. In our case- the expert part should interrogate
the specific employees regarding safety aspects of an enterprise.
The next figure illustrates how a safety expert, with the
necessary knowledge into the problem could impr…