Mike on Security

Post navigation

FindBugs is similarly configured for multi-module Maven builds as the Checkstyle and PMD plugins. As in the Checkstyle and PMD example, the configuration example below shows how to set up a “developer build” as the default configuration, allowing a developer to execute FindBugs with the “mvn clean install” command and create data for reports instead of failing the build. As usual, we will use a Maven profile for release builds, so that developers can create local reports, and the CI/CD system can fail the build in case of violations.

The benefits of using a centralized FindBugs filter configuration are in my opinion somewhat limited, but it is still possible to create and use such a configuration artifact (jar resource file) in the Maven FindBugs plugin configuration. The “dependency” section pulls in the configuration artifact, and the “excludeFilterFile” parameter references the FindBugs rules configuration in the resource configuration artifact. This filename must match the actual rules filename used in the dependency. The configuration settings are parameterized with Maven build variables, but this example also works with hardcoded settings.

There are two major options to automatically add license headers to source files:

Mojo License Maven plugin from codehaus

Mycila Maven License plugin from mycila

I think that the “mycila” plugin is easier to use than the codehaus plugin. However, it seems as if the codehaus plugin is the go-forward tool, despite its problems and quirks. I found it very difficult to make the mojo Maven license plugin work in a way so that I could use my own license file in a centralized configuration artifact (i.e. a dedicated jar file holding the custom license file).

The good thing about the mojo Maven license plugin is that, in addition to the source code license headers, it can also create (although unfortunately not reliably update) the project license file.

As a first step, add a “license” section on the project level to document the license choice. It is commonly recommended to use a compatible name and identifier for the license. For open source projects, most licenses already have an SPDX name identifier assigned.

Once the license is defined and chosen, configure it for the project as follows:

Create a centralized configuration artifact, with a “resources” directory containing a “licenses” parent folder, and within the “licenses” folder, a folder named with the license identifier (e.g. the SPDX identifier). Only use lowercase letters for this directory name, and do not use spaces. This directory contains two files: One for the license header that goes into each file (“header.txt.ftl”), the other one for the license that goes into each Maven module’s root folder (“license.txt.ftl”).
The license header file contains the body to be used in a generic template that comes with the plugin, while the license file that will be placed into the Maven root folder actually is a template that allows substitution. The content from the header file will later on be completed with a package description line and a copyright line, while the Maven root folder license file will not go through this transformation. For some reason, the header file cannot contain blank lines, as blank lines in this file will lead to a misleading error message.

Create a “licenses.properties” file in the license parent folder in the resources folder created in step 1. This file must contain a mapping of the license folder name (aka the license name or SPDX identifier) to a human readable name. This makes the license available to the plugin (the name is displayed, referencing the identifier, which by convention identifies the folder that holds the license files).

Build and release the configuration artifact

Configure the license maven plugin. Reference the folder name in the resource folder of the configuration artifact from step 1 as the <licenseName>. The <licenseResolver> tag must contain the path to the folder from step 1. As an example, if the license is stored in ./src/main/resources/license/mylicense, then “mylicense” is the licenseName and “license” is the licenseResolver. Include the dependency to the configuration artifact as a dependency of the License Maven plugin, and complete the configuration of the plugin as needed (e.g. if it should fail the build in case of missing / incorrect licenses, if the plugin is allowed to modify the license section in files, which files to ignore, etc).

Add the following required project meta information:

<inceptionYear>
This is used in conjunction with the current year as the value for “${copyright.years}” in the Maven module folder license file, and it will automatically be inserted in the file headers before the body of the header license notice

<licenses> with <license> block containing name, URL, and distribution (set to “repo” in this example)

Name and URL of the organization holding the copyright
This is used as the value for ${copyright.holder} in the Maven module folder license file, and it will automatically be inserted in the file headers before the body of the header license notice after the “${copyright.years}” value

For each Maven module, add a ./src/license directory, with a “licenseDescription.ftl” file. This file should contain a single-line module specific description, which will be prefixed to all headers in the respective module. The content of this file is added into the file header right before the copyright years, copyright holder, and header license body.
This file must not contain any blank lines. Some IDEs add a newline at the end of this file when editing / saving it. The license plugin removes this newline when it updates the license headers (good!), but it is not smart enough to also do this during the subsequent checks when it tries to find out if a file header needs to be changed. Consequently, every subsequent build will fail with an error of “file headers need to be updated”, but an update of the headers will not fix the issue. The solution is to remove the newline in this template file.

This is how the overall configuration in the parent POM should look like. Note that the name in the organization contains an html entities encoded representation of the email address to make it work with the License Maven plugin. Some of the configuration values are parameterized, but the example works equally well for hardcoded values:

It should normally only take 10-15 minutes for an artifact to propagate to Maven Central after the staging repositories have been released in Sonatype OSSRH. If the artifacts do not show up on the browse view at http://search.maven.org/#browse, then there might be something broken with the index that powers this view.

I asked Sonatype support about this, and luckily, the artifacts are still available in Maven Central, even if they do not show up in the search index on search.maven.org. The search index and the browsing view are, in fact, completely decoupled from the actual storage backend, and http://search.maven.org is regularly updated about 1 to 2 hours AFTER an artifact has been pushed to Maven Central.

If an artifact does not show up in the browse view on http://search.maven.org, browse the Maven Central repository directly at http://repo1.maven.org/maven2/. Artifacts should show up there within 10-15 minutes after doing a release. If they are not visible after 20 minutes, the deployment most likely failed.

In this series of posts, we discuss how to publish Maven bits to Maven Central using the standard Maven deploy plugin. This is an eight step process to set up and execute, and we discussed the first two steps in the first post of the series and steps three to five in the previous post.

In this post, we discuss the last three steps with particular focus on how to do the code signing with GnuPG in an automated multi-module Maven build.

Step 6: Configure GnuPG for artifact signing

Neither Maven nor Nexus can properly handle GnuPG subkeys as signing keys. Using a subkey for signing is actually a very good security best practice, because it allows protecting the main key in automated code signing setups. Unfortunately, due to the limitations imposed by Maven and Nexus, if you are used to subkeys signing, then you must change your habit to make GnuPG work with Maven.

It also seems to be impossible to specify the singing key in the Maven GnuPG plugin. Consequently, you must delete not only all of your signing subkeys from the primary (master) key, but also all the other secret (master) keys you may have on your key ring. Once you deleted all secret keys save one, and all signing subkeys of that remaining primary key, the build should produce valid signatures.

For the basic Maven configuration, add the following section to the parent POM, preferably in a dedicated profile so that the signing is only executed when needed (e.g. on a centralized build server):

As GnuPG keys are usually protected by a passphrase, it is required to somehow inject the signing key’s passphrase into the Maven build. There are three common options for this:

The passphrase can be stored in a settings file.
This is really a bad idea, in particular as Maven requires using the master key, and is incapable of operating on an easily revocable subkey.

The passphrase can be submitted as a parameter when starting the build.
Use the -Dgpg.passphrase=keyPassphrase parameter when starting the Maven build. This is a reasonably safe option on single users systems (e.g. on dedicated build servers), but may allow users on multi-user systems to steal the passphrase from the process list. For multi-user systems, the parameter can point to an environment variable so that the passphrase does not show up in the process list (this is really a must on multi-user systems).

Provide the passphrase during the build when it is needed
The plugin will ask for the passphrase if it cannot find it in a file or as a parameter. This is the most secure option, but it may prevent build automation if not properly configured. By default, GnuPG uses its agent to retrieve the password, and the default setup is an interactive prompt. Depending on how the pinentry component of the agent is configured, it can retrieve the password interactively or non-interactively e.g. from a password vault.

Step 7: Build and deploy to staging

Build the binaries according to the project’s release process. Assuming that all the release related plugin confirmations are in the “release” profile, such a build could look similar like this:

The build instructions (and POM configuration in the underlying project) of Mike’s utility library give an example on how to configure an enterprise style Maven multi-module build and set it up to automatically deploy to Sonatype Nexus.

Step 8: Release the deployment to Maven Central

As indicated earlier, the automated deployment to OSSRH with the standard Maven Deploy plugin does not automatically release the deployed artifacts to Maven Central. While this process can be automated, I generally prefer to review the build results once more before releasing them to the public.

Sonatype describes their release process in detail. Once the deployment has been released, Sonatype will do an (automated) evaluation of the staging repository contents, and promote to Maven Central if everything is okay. Otherwise, the process will fail and the (then obsolete) staging repository can be “Dropped”.

We will discuss the five of the remaining steps in this post, and I will create another post with particular focus on how to do the code signing with GnuPG in an automated multi-module Maven build.

Step 3: Request an account in Sonatype JIRA and create a ticket to set up a Nexus repository

The process of how to request an account and have a Nexus staging repository set up is described in the OSSRH guide. The staging repository is restricted to a specific group ID. If you maintain multiple projects and eventually want to publish all of them, request a top-level group ID in the ticket.

As an example, if you own the domain “domain.com” and you have two projects with the planned group IDs “com.domain.project1” and “com.domain.project2”, request the repository to being set up for “com.domain”, even if the concrete request is for the project with the group ID “com.domain.project1”.

Step 4: Add the OSSRH repository to the distribution management section in the POM

I do not publish SNAPSHOT builds in my project, but Sonatype generally supports them. They are located in a different repository though.

Configure the distribution management section to deploy the bits automatically into the staging repository on Sonatype using the standard Maven deploy plugin:

See the “util” project parent POM and related source file here for an example.

Step 5: Add your credentials for OSSRH to the “settings.xml” file in your home directory

The Maven deploy plugin will need the Sonatype jira credentials to submit the bits to the staging repository. It is really not very secure to use the primary credentials directly here, but unfortunately Sonatype does not support using a tokenized form of the credentials at this point.

Be sure to always use the same ID whenever referring to the OSSRH repository: The ID used in the repository ID of the “distribution management” section of the POM must match the ID used in the settings’ server section.

There are several automated options to publish artifacts that have been created in a Maven build to a Maven repository. The deploy process may depend somewhat on the repository being used. For example, if a project is not hosted by one of the big players, it might be necessary to deploy through a “proxy” mechanism that eventually forwards the artifacts to Maven Central, rather than deploying there directly.

One of these proxy services is provided by Sonatype as their “OSS Repository Hosting” or “OSSRH” service. The OSSRH service provided by Sonatype is free, and allows staging of one’s artifacts and then promoting them to Maven Central.

Sonatype provides detailed instructions on how to use the service, heavily promoting their “Nexus Staging Maven Plugin” for Maven based builds in combination with the Maven Release plugin. While there is nothing wrong with using these plugins, I find it hard to integrate into enterprise builds where the build process is a little more formalized and a well-defined branching and tagging process should be obeyed.

Following the eight steps outlined below, projects can be deployed to Maven Central using the standard Maven deploy process. The client side (i.e. the Maven build, tagging, branching, etc) can still be easily automated by employing a build system such as Jenkins, giving the build engineer tight control over how the build is performed. However, once the artifacts have been deployed, it is still necessary to “promote” them from the OSSRH Staging Area to Maven Central. While this step can be fully automated when using the Nexus Staging and the Maven Release plugins as recommended by Sonatype, I do not mind this last manual step and find it useful to be able to review the staged artifacts before they are published to Maven – in particular as releases are not that frequent. On the other hand, if a project uses OSSRH to also publish SNAPSHOT artifacts, which are commonly released with a much higher frequency than regular releases, it might be prudent to either follow a hybrid approach (using the staging plugin for SNAPSHOTS, and the regular Maven process for other releases), or to adopt a project workflow that fits in the boundaries defined by the Nexus Staging Maven Plugin.

The examples below are based on a utility library that I published through this process and commonly use as a Maven dependency. This particular project is a multi-module Maven build.

Step 1: Create a GnuPG key pair, and publish the public key

Sonatype has an excellent guide on how to generate and manage keys with GnuPG. They are very clear about the requirement that the key used to sign the artifacts cannot be a subkey. This is not very desirable from a security and key management perspective, but unfortunately necessary due to limitations of Maven and Nexus, which can only verify signatures against a primary key.

Step 2: Set up the Maven build so that it meets the OSSRH requirements

I noticed a few things than merit additional clarification in addition to what is mentioned in Sonatype’s guide:

The build must produce a source and a Javadoc package in addition to the artifact. This is tricky in case a project is a mere resource project, that is, a container for resource files. In this case, it will usually not contain Javadocs. However, as Sonatype requires Javadocs to be present, it is possible to satisfy this requirement by creating a “fake” Javadocs file. This can be done by creating a src/main/javadoc directory and adding a README.txt file. The fake Javadoc can then be included by adding the plugin below. See the “build-tools” project POM and related source file here for an example.

The group ID must be owned by the project. For example, a project cannot write to the google.com group ID unless it is authorized to do so. If necessary, reserve a group ID for the project when requesting access to OSSRH.

The parent POM of the project must contain a “developers” section, but that section does not necessarily have to contain an email address that is trivially machine readable. You may try to protect your email address against UBE, like in the example below.

I recently published a small project on GitHub, mostly to play around with the features that GitHub offers over plain git. I came across the GitHub pages features, and wanted to try creating a site from my multi-module Maven project, and have that automatically published on GitHub pages.

It turns out that building a Maven site is a very fragile process, in particular for multi-module builds. There are countless blog posts and posts on stack overflow that deal with this problem, and I am adding just one more after spending half a day figuring things out. I just could not get the internal project links to work: Either the link worked from the direction of the site created by the parent POM to the child POM, or the other way around. At some point, I got the links working in both directions for the site’s side menu, but not for the links in the content section (e.g. the Project Modules overview page).

The key to the problem lies in how Maven creates the relative URLs in the project. Obviously, the project needs a “distribution management” section for the parent module (aggregator project) of the multi-module build. It also needs a “distribution management” section in each child module for the links to work. The URL in each of these sections has to be different in each project, otherwise the links in the report are broken. The reason for this is that some of the relative links in the generated site are computed based on the URL from the “distribution management” section.

In addition to the URL in the “distribution management” section, each module must also have a top-level URL parameter, because some of the links are computed relative to the URL from this top-level parameter.

To make things extra tricky, the two URLs must be the same, because otherwise the two algorithms that compute the links produce a site where some of the links work, and others do not.

As an example, I incorrectly used one of the links as a pointer to the project’s start page, and the other one as the pointer to the project’s Maven page. They shared the same domain, which probably contributed to the mess: The index.html page that the project-report plugin produces (“project information” -> “about” page -> “project modules” overview section) uses the main project URL to derive links, which all ended up having a “../” too much in the relative URL. The maven site plugin uses the “distribution management” URL in “mvn site:stage“, which resulted in the same links being correct in the “modules” side menu.

It is crucial to set the project URL correctly on each level, and then reference a module’s project URL again in its distribution management section to overwrite anything that may be inherited from a higher level POM.

The id in the “site” section may have to match an entry in the ~/.m2/settings.xml file, depending on how the site is being distributed (e.g. automated upload to a webserver). For the setup on GitHub that I am using in this project, this is not required (I decided to copy locally, and then commit to the gh-pages branch – much less brittle, and not a lot of extra work).

Note that the format of the URL must be specified as in the example, and the distribution management section must be present.

The distribution management section in the child POM looks exactly like the one in the parent POM. However, if it is omitted, then the distribution management section of the parent will eventually be used. It took me a while to figure this out, but apparently the order in which Maven seems to process this is to first do variable expansion, and then inheritance second. This means that, if the distribution management section were removed in the child, the child POM would inherit the parent’s distribution management section with the expanded URL, putting the value of the parent’s project URL into the child’s POM, and not the child’s URL that contains the artifact ID. Or, in other words, the URL value in the child POM would be ignored.

The interesting thing is that, according to the Maven Site Plugin documentation, the inheritance should work correctly as long as the multi-module build follows the Maven naming conventions:

If subprojects inherit the (distribution) site URL from a parent POM, they will automatically append their artifactId to form their effective deployment location. This goes for both the project url and the url defined in the <distributionManagement> element of the pom.

If your multi-module tree does not follow the Maven conventions, or if module directories are named differently than module artifacts, you have to specify the url’s for each child project.

As my project follows the standard naming convention, and removing both the <url> and the <distributionMananegemt> sections from the child POM leads to erratic behavior with broken links, the “automatic appending” is apparently not working correctly.

The BUILD.md file contains instructions on how to perform a release build using this setup. This in particular includes automatically (and consistently) setting the version for all POM files in the multi-module build, building the site, staging it locally, and then publish it on GitHub pages.

Note that it is recommended to always build the project before building the site, as the site goal may or may not report missing dependencies if it does not find them. A possible symptom of this problem is that the Javadoc plugin will not find your classes if “mvn install” is not run before “mvn site“:

I do not use a plugin to automatically publish the site on GitHub, but instead copy it manually from the staging folder in the “target” folder to the appropriate folder in the “gh-pages” branch of the same repository. This requires me to check out the project twice on my machine, once on the main branch, and once on the “gh-pages” branch. I still like this approach better than automating it, because building the site for multi-module builds seems to be very brittle in Maven, and this gives me the opportunity to review that everything has been built correctly (and all the links work) before I publish the site with a git commit. Also, while there are a few plugins that automate this process, such as com.github.github:site-maven-plugin and other plugins that implement wagon for GitHub in Maven, none of them seem to be able to properly deal with multi-module sites at this time.

I also use a small project intro page on GitHub pages with some basic project information and links to relevant documentation (e.g. the Maven site, and the Javadocs). The page I used there is a GitHub standard template. It looks good enough for my needs, and offers convenient links to download the project source code as a zip file or tar ball.

I have recently been asked a lot about the NIST process mentioned in an earlier post. In a nutshell, NIST was working on a paper describing (software) security as a holistic approach, deeply embedding a security mindset into traditional systems engineering patterns with the goal of building secure and resilient systems, instead of bolting security on in a later stage of the game.

NIST has meanwhile published more current draft materials (http://csrc.nist.gov/publications/drafts/800-160/sp800_160_draft.pdf ). At this time, the link points to a draft dated May 2014. The draft was released about six months ago, and NIST 800-160 starts to pick up with the industry. I had numerous inquiries from major HP customers asking questions on HP’s internal security processes in context of NIST 800-160, and how HP is proactively dealing with security threats not yet known while the product is being built. The language of the requests with terms such as “resiliency”, “trustworthiness”, and “build security in” strongly resemble the language NIST and the Department of Homeland Security have chosen in their publications around software that should be able to not only withstand known attacks, but also be ready to resist new threats or possibly even actively defend against them.

John Diamant is an HP Distinguished Technologist and secure product development strategist in HP. He has done a lot of work on how to design security into a product, and with HP’s Comprehensive Applications Threat Analysis (CATA), John has created a great toolkit to automate parts of the security requirements and threat analysis.

John is working a lot with the US public sector, and he certainly sees a lot of the feedback HP receives around the NIST and DHS security initiatives. He has some very interesting comments on how to create secure and resilient software, and how a Secure Software Development Lifecycle (SSDLC) program fits into this: http://www.securitysolutionswatch.com/Interviews/in_Boardroom_HP_Diamant2.html

This is the last post in a series of posts where we discussed a few Secure Software Development Lifecycle (SSDLC) metrics that I personally find very interesting. The four metrics are:

The number of qualified defects found per quarter

The number of qualified defects fixed per quarter

The difference between [1] and [2], that is, inflow vs outflow

The overall number of open qualified security defects

In this post, I will share the last metric of the series, which is a graph of the overall number of open qualified security defects, and their development over time. This data is taken from the same organization and product set discussed in the first post of this series, and it has been anonymized, randomized, and transformed into a relative scale to protect confidential information without affecting the trends that are typically seen when starting a new SSDLC program. All percentages in the graph are relative to the results of the 20 Quarters before the SSDLC program was started.

Overall Number of Open Qualified Security Defects

The graph below shows how the overall number of open qualified security defects developed in the example organization. All percentages in the graph are relative to the results of the first 20 Quarters. The organization started out from a backlog of 100% of open issues, and continuously added to that backlog over the following 9 quarters. The backlog peaked at 242% of open issues, and then started to decrease slightly. This means that R&D teams more than doubled their security defects backlog, despite the great increase of fixes they released in the same time, which is a very impressive achievement.

The graph shows that the teams did a good job of keeping the number of critical issues in the backlog consistently low, and even managed to significantly reduce the number of open critical defects in Quarter 9.

We can also see from the graph that in Quarter 9, the teams managed, the first time in more than 7 years (all for which data is available for this organization), to reduce the size of the overall security backlog. This is obviously very much aligned with the inflow/outflow metric, where Quarter 9 shows a negative inflow (or a net outflow) of security issues in the backlog. This is a big achievement, and a good indication that the SSDLC program was solidly embraced by the organization’s senior leadership as well as by the organization’s engineers.

*Important: Note that an organization maintaining a security issues backlog does not necessarily mean that the organization releases products with known security vulnerabilities. Companies such as Hewlett-Packard have strong business ethics, and do their best to protect their customers from all known and unknown security risks. If a security defect in the backlog affects a version or platform that has not yet been released, or may manifest itself as a vulnerability only under certain operating conditions that are not present in supported setups, then a security defect may be backlogged without increasing the security risk for the customer.

For example, a library that is being used in a product may have a defect in input validation, thus leading to a buffer overflow in the library, ultimately affecting the product using the library. The SSDLC process would discover such an issue and track it as a security defect in the backlog. However, the “supported setup” for this library may be that it is only to be used in a product that performs its own input validation before passing any user provided data to the affected library method. As long as the library is being used in this “supported setup”, there is no security vulnerability in the final product, which means that the security defect in the library does not result in a security vulnerability and hence does not translate into a security risk for the customer. Still, the security defect of the library is being tracked in the backlog, so that a fix for the library can be obtained, or the library can be replaced.

Next Steps

One goal of an SSDLC program is to reduce risk and increase confidence that security issues are properly handled. Backed by this confidence in the SSDLC process and the quick turnaround times on security issues an organization will eventually achieve, the leadership team may define new Service Level Agreements (SLAs) for open security issues.

For example, an organization may establish strict SLAs with time windows in which security defects would have to be fixed (issue qualified, defect fixed, patch tested, patch delivered / publicly available). The organization may split this up in a matrix for low / medium / high / critical severity and the source of the defect (internally found / reported by customer under NDA / publicly known). Ideally, the organization should define another metric on how well they delivered against these SLAs, making them true security professionals!

In the first post of this series, we discussed a few Secure Software Development Lifecycle (SSDLC) metrics that I personally find very interesting. The four metrics are:

The number of qualified defects found per quarter

The number of qualified defects fixed per quarter

The difference between [1] and [2], that is, inflow vs outflow

The overall number of open qualified security defects

In this post, I will share some inflow / outflow metrics, and their development over time. This data is taken from the same organization and product set discussed in the first post of this series, and it has been anonymized, randomized, and transformed into a relative scale to protect confidential information without affecting the trends that are typically seen when starting a new SSDLC program. All percentages in the graph are relative to the results of the 20 Quarters before the SSDLC program was started.

Inflow vs Outflow

The inflow / outflow metric gives a good indication on how successful an organization is in dealing with newly found issues: can they qualify and address security defects fast, or are they overwhelmed by the influx of new issues, and just keep piling them onto a backlog?

This graph shows the difference between the number of incoming new defects and the number of defects that have been closed in the same time period. Like in the previous illustrations, the graph shows relative numbers (percentages) in relation to the results of the first 20 Quarters. Unfortunately, this makes the graph a little harder to read, because the percentages do not directly translate into the actual number of issues that have been added to the backlog. In this graph, big positive percentage numbers mean that work is added to the backlog. A negative percentage number (or a number close to 0) is desirable, because this means that the team is ahead of or at least keeping up with the influx of work.

This graph shows two peaks, which is a very common characteristic for organizations where multiple groups contribute to finding security problems, but only one group is fixing them. If we compare this to the two graphs we discussed in the previous post, we can explain the first peak by a large inflow of defects, which the developers are not yet prepared to handle. After about 12 months into the program, the engineers are starting to catch up. This is very common, and a good sign, because it reflects the learning and training that the developers have to go through to know how to deal with the findings, and then the developers applying these learnings and starting to catch up with incoming work. This first peak is usually always present when rolling out a new SSDLC program.

The second peak is very typical for an organization with dedicated QA teams or security pen testing teams. Once these teams have been ramped up, completed their training, and are fully operational, their reporting output picks up significantly – typically also after 12 months. Other than the R&D team (who may also report more defects against their code), they are usually not chartered to fix the issues they discovered. This leads to additional pressure on the developer team, and the developer team must adjust (again) to the higher influx of security issues. Once this adjustment is complete (Quarters 8 and 9), the organization reaches an ideal state of close to zero or negative inflow / outflow difference.

The graph also reveals how an organization is prioritizing work. In this case, the organization is rightly prioritizing work on critical security issues. However, we can also see that they are focusing on medium and low severity problems first, before addressing high severity issues. This may be justified, for instance if crucial resources who can deal with the high severity problems are not available as they are assigned to work on the critical severity defects, or if some of the higher severity defects are not manifesting as security vulnerabilities (e.g. due to defense in depth covering them), allowing delays to fix lower-severity problems that do lead to security vulnerabilities. This metric makes potential prioritization problems visible and actionable.