When should we scan for vulnerabilities in our build using WhiteSource Bolt?

Let’s discuss a few experiences from recent evaluations and hopefully trigger a vibrant discussion. Are you using WhiteSource Bolt or WhiteSource? If it is the former, please read on.

Revision

Date

Note

1.0

2018.08.15

Revert to single compressed zip package file

2.1

2018.08.16

Add feedback from Jesse Houwing

2.2

2018.08.29

Add ignore (exclude) option

3.0

2018.09.07

Add decision chart and revise recommendation

What's WhiteSource?

WhiteSource integrates into the build process and helps you detect open source components and manage open source license compliance, security, and vulnerabilities. With OSS security vulnerabilities appearing on the OWASP Top 10 vulnerabilities, its pivotal to inform all stakeholders of potential security issues before we deploy to production.

Proof of concepts (POC)

We evaluated three simple build scenarios on Visual Studio Team Services (VSTS):

Run the WhiteSource Bolt scan against the complete solution before the build, after setting up NuGet and NPM (recommended by WhiteSource).

Run the scan against the complete solution after the build and after deleting or ignorting (exclude) the nodes_modules folders, which contains all the npm libraries and its dependencies.

Run the scan after the build, against all the build artifacts which will be used by the VSTS release.

Generates a report with thousands of scanned libraries, with many false positives. Reminds me of the NPM craziness we had with Ranger extension pipelines. It is simply too much noise for the developers. They will start ignoring the scan reports, which negates the value of the security scan.

However, as noted by Jesse, the advantage of having the scan early in the pipeline is that you can catch malicious build-time payloads. While you do get a lower noise later in the pipeline with 2 & 3, you miss out on an entire set of potential attack vectors created by build-time dependencies. You could be leaking credentials, vulnerabilities could harvesting values or adding malicious code into your package management, and much more.

SCENARIO 2

Reduces the report to hundreds of scanned libraries. Still a lot of noise for the developers.

Instead of deleting the node module files, we can configure WhiteSource Bolt to exclude (ignore) the folders, as shown.

We have reproduced an issue with the WhiteSource Bolt exclude feature. If you are using the task v18.8.2 or lower, you must use the backslash when defining the exclude path.

SCENARIO 3

WhiteSource Bolt has some limitations on the kinds of analysis it can run. For example, it is unable to scan compressed/zip files. We therefore copy the $(build.artifactstagingdirectory), extract the compressed files, and then scan, as shown below. This limits churn and risk, as existing build and release pipeline tasks are not affected.

Copy the $(build.artifactstagingdirectory) folder

Extract zip files

Run Whitesource against unzipped folder

Drops the noise to 10x actionable libraries. Great!

NOTE

All three scenarios report 10 vulnerabilities. The reports only differ in terms of the number of scanned libraries with no vulnerabilities.

But, is scenario 3 really great?

We all agreed that the drop/staging folder contains what matters most. But, I have not slept well over the past few days, pondering about scenario 2 and 3.

Are we potentially missing vulnerabilities?

Can WhiteSource Bolt scan minified Java Script files to determine what packages were used for a Java Script application?

Here's some feedback from fellow Rangers and WhiteSource:

"For the false positives, it is very annoying but for cost most of my teams using this just live with it. The thing they care most about is the report with the list of 3rd party libraries, vulnerabilities, and OSS licenses that goes to the CSO or CIO. For that we export to Excel and annotate that report to eliminate the false positives and explain why." - Mike Douglas, ALM | DevOps Ranger

"As far as reducing the noise – at the moment, it’s the placement of the task in the pipeline and trying to stage the actual artifact contents to enable a scan to be as effective as possible and to eliminate unused dependencies and content. Obviously, that’s much easier with a .NET Core project than with a JavaScript project." - Ken Muse, ALM | DevOps Rangers

"Bolt does identify all the Nuget and Node packages, source files (like js files), and all their vulnerabilities. If a minified JS file is part of drop/staging folder and it is part of the CDN repositories WhiteSource scan, then it will be picked up by Bolt." - Yossi Weinberg | WhiteSource

So, which scenario should you use?

Scenario 2 and 3 were created to address the high-noise and potential false positives when working with NPM (node modules) based solutions. If we take a step back, we can ask a few questions to pick the right scenario:

Always combine scenario 2 and 3 with a scheduled scenario 1 scan, to catch potential attack vectors created by built-time dependencies, such as Leaking Credentials.

Food for thought

As part of your evaluations, review the full WhiteSource product as well. It has some more advanced identification features, such as:

Real-time alerts

Ability to scan inside archives

Identify js component in html

No need to identify the actual file in the file system, it can deduct it from html script attributes