Measuring The Effectiveness of Security Awareness Training

Online security awareness training is now the most popular form of security awareness training in the world. As we noted here, that’s good news when it comes to measuring the effectiveness of security awareness training.

Offline, things aren’t so easy to track. However, online it’s possible to see who is doing what where and when. It’s little surprise, then, that measuring the effectiveness of online security awareness training has been chiselled onto the agendas of CISOs for some time.

And yet, measuring the effectiveness of security awareness training remains difficult. In this post, we’re going to go through how companies typically track the effectiveness of their security awareness training and where they may be going wrong, before suggesting an alternative approach.

How companies measure the effectiveness of security awareness training

As far back as 2012, Educause Center for Analysis and Research were looking into measuring the effectiveness of security awareness training. In 2013, the Center published the results of a survey that asked 95 universities how they measured the success of security awareness programs.

Reported methods of measuring the effectiveness of security programs included:

Monitoring the number and type of security incidents they experienced (62%)

Employee feedback (45%)

Monitoring behavioural change (34%)

Surveys (24%)

Training attendance (24%)

Alignment with university strategic goals (9%)

Targeted assessments (9%)

Pre- and post-testing to monitor performance (8%)

Given the survey took place in 2013, the reported efforts are certainly commendable – especially as it appears at least some research participants were measuring the impact of their security awareness campaigns in more than one way.

The right way measure the effectiveness of security awareness training

As revealing as the above survey is, it’s probably not revealing enough. Measuring the effectiveness of security training requires more than monitoring a selected set of metrics.

Step 1: Strategy

A coherent and well-planned cyber security strategy is a prerequisite for measuring the effectiveness of cyber security training. Without a strategy, any delivered security awareness training runs the risk of becoming a fleeting effort.

So before thinking about measurement, think about what you want to achieve and how you want to achieve it.

At CybSafe, we advocate keeping a close eye on three things: security awareness, security behaviours and security culture – the ABC of information security.

For us, that means CISOs need to focus on:

What people know and understand about how to stay safe online (awareness)

How people really behave when presented with attacks (behaviour)

What people think about – and how much they care about – cyber security (culture)

And how confident people are in their abilities (culture)

So, back to locking screens. To improve the incidence of screen-locking, what is it that’s really important to find out?

On the awareness front, you may wish to find out if people know about risks and best practices. When it comes to behaviours, you’ll presumably want to find out how likely it is for an unattended screen to be left unlocked. When it comes to culture, you’ll probably want to find out why people are behaving in the way they are.

Step 3: Metrics

Measuring awareness

The above questions help highlight useful metrics to monitor. Take what people know and understand about how to stay safe, for example.

People’s knowledge and comprehension of security can be monitored through online security awareness training performance. So long as you have access to a cyber awareness platform with analytical capabilities – such as CybSafe – it becomes easy to see how much people know about security best practices. Online quiz results can reveal whether or not people know of the risks of leaving unattended monitors unlocked. And that’s at an organisational level, a departmental level and an individual level.

Measuring behaviour

Measuring behaviour, meanwhile, is usually best achieved through simulated attacks.

Simulated attacks test the security behaviours of the people in your organisation. Monitoring how people respond to simulations gives you a metric of security behaviour.

Simulated attacks may not be appropriate when attempting to measure the chances of an unattended computer screen being unlocked at any given time… but a coherent strategy makes finding alternatives easy. Spot checks, for example, should give you the information you need. You may consider combining checks with an incentive system, perhaps using yellow cards for unattended screens that have been left unlocked, and chocolate bars for screens that have been locked!

Indications of behaviour change can also be measured in other ways, for example measuring triggers and motivations – two key components widely acknowledged as necessary for behaviour change to occur.

Measuring culture

Measuring culture is perhaps the hardest of the three to do – but it’s not impossible. Anonymous surveys, for example, can give you an idea of why people take risky actions like leaving their screens unlocked. And they can be revealing.

Perhaps it’s an unwritten rule that it’s OK to leave screens unlocked when heading over to the printer. In theory, it’s low risk. But should the printer unexpectedly need more paper, risks increase.

Qualitative insights from surveys can help you change behaviours and reduce risks – but it’s important to note that finding an overall quantitative cultural metric is key. It’s only through quantitative metrics that improvements can really be pursued.

Through surveys, employee feedback and the intelligent analysis of sentiment and attitude, the CybSafe platform measures culture in both qualitative and quantitative terms.

Step 4: Timing

When measurements are taken is also particularly important.

To measure the effectiveness of security awareness training, measurements should ideally be taken at day zero – before security campaigns begin. Then, as campaigns unfold, measurements need to be recorded at regular intervals – ideally monthly, or even more frequently.

While this might be problematic when attempting to measure the impact of security awareness training manually, intelligent platforms like CybSafe naturally log awareness, behaviour and culture metrics automatically, allowing CISOs to conduct analyses as necessary.

As well as fuelling continuous improvement, regular measurement helps identify where people may need further support, and administering further support can ultimately prevent breaches.

Unfortunately, as the metric fails to take into account the number or severity of attemptedattacks, the statistic can be misleading. If the number and severity of attacks increases while training is delivered, the number of incidents experienced may well increase regardless of how effective security training is.

Measuring fuels machine learning and AI

At their heart, machine learning and AI are both fuelled by measurements.

Machines measure, then trial something, then measure again.

The difference between the two measurements is what allows machines to learn. It’s what makes AI possible – and it’s how our intelligent security awareness platform is able to evolve with the needs of your organisation.

CybSafe combines applied machine learning with cognitive computing technology to learn which actions to take to decrease your organisation’s human cyber risk. It’s still measurement – just automated. Without measuring the right things, the innovation wouldn’t be possible.

Why measuring the effectiveness of security awareness training is important

Historically, measuring the effectiveness of security awareness training has proven problematic. So problematic, in fact, that some of today’s top CISOs are unable to measure the effectiveness of their security awareness training as well as they might like.

Fortunately, the rise of intelligent cyber awareness platforms is making measuring simple. Those who wish to measure the effectiveness of security awareness campaigns are advised to give them proper consideration.

After all, monitoring the effectiveness of security awareness training is essential when it comes to truly reducing an organisation’s human cyber risk.