Software Integrity

What story do your mobile metrics tell?

As people become more reliant on their smartphones, mobile applications become an important focus for many organizations. There are many articles about adapting your software security group (SSG) to handle the new risks posed by new technology. But, are you confident that you are tracking your organization’s progress and performance effectively? What story do your mobile metrics tell? Are you confident that you are able to show the impact your SSG has when addressing mobile security?

Metrics optimization

A useful security metric tells a story around the impact and value the SSG adds to the organization at large. Rather than reporting how many mobile applications your organization has enrolled in dynamic scanning, there is more value in reporting how many high severity findings your dynamic scanning discovered and/or how many findings were remediated due to the scanning efforts. This shows the impact of dynamic scanning on the quality of the code.

A good metric has the following attributes:

It asks a question. For example, are revenue-generating development teams employing penetration testing to discover risks?

There is a defined goal associated with it. For example, understand how much unknown risk exists in the organization’s application portfolio by discovering which applications have and have not been penetration tested.

Reliable and consistent data points are used to calculate the metric. For example, the number of applications penetration tested over a certain timeframe, and the number of applicable applications in the same timeframe.

The metric calculation consists of more than a single number. Ratios can convey more meaningful information. For example, the number of application penetration tests / number of applicable applications.

Even though I say that there’s more to security metrics than those simply describing adoptive efforts (such as how many mobile apps are enrolled in binary analysis), sometimes it’s okay, and even appropriate, to have adoptive metrics. This works best for SSGs that are new and working to build more maturity into their mobile software development life cycle. However, SSGs should avoid getting stuck in this mindset. It is important to re-evaluate the metrics SSGs are reporting, to introduce metrics showing the effectiveness of the activity, and the impact of the security organization as soon as possible.

Here are three examples of impactful metrics that your organization can leverage:

1. Prevention of defects with a ‘top N’ security bug list

Tracking defect density is a great way to get an overview of the persistent risk within your organization. However, it fails to give you the specific information needed to determine how to address the problem. If you track defect information and categorize defects by type, you can understand what the top N security bug types are within your organization. With this information, you are now able to understand how to focus your efforts to best reduce risk within your organization. For instance, if cross-site scripting (XSS) is the number one defect in your organization, now you know that creating a campaign to squash all XSS findings by providing specialized training to identify and remediate XSS vulnerabilities is worthwhile. This training can then be provided to the application teams (typically the worst offenders) or the global organization as a part of developer training.

2. Mean time to remediate defects

Let’s say that your organization has mature bug tracking solutions that are integrated with both the application development teams and security testing teams. In this case, it is possible for your organization to track the mean time it takes to remediate defects. By tracking a timestamp noting when the security team reports a vulnerability, and another timestamp noting when the application development team resolves the vulnerability, it’s possible to understand how long it takes for security issues to be remediated by your organization. However, without mature bug tracking solutions, this is a very difficult metric to track.

3. Impact of security controls

It’s possible to show the impact of the security controls that your organization leverages. It might take some time and creativity to define the metric to accomplish this goal; but by doing so, it will enable your organization to make smarter, more informed decisions which will only increase the effectiveness of your team.

The sooner you can create impactful metrics, the better you can equip your organization to create smart goals.