Tuesday, December 26, 2006

The future of web application vulnerability assessment is about scale

Recently Alan Shimel (StillSecure) went out on a tiny twig and said, “vulnerability assessment (VA) is dead”. Of course Alan’s speaking about network security not web applications. His remarks are about VA's convergence with NAC’s. Fair enough. When I spoke with him he said, “Actually VA for web apps is one of the few bright spots in the VA space these days.” I'd like to think so. :) This topic is always on my mind since this is exactly what my company does. “What is the future of web application vulnerability assessment?” is a question that doesn’t get asked a lot. Personally I think we’re at the point where network VA was a few years ago, solving the challenge of scaling.

Granted my numbers could be off and may vary a great deal from enterprise to enterprise. However, this exercise helps estimate the relative needs of the market. Let's see what kind of resources we need if we're trying to assess all these websites for vulnerabilities twice per year.

Today we'd need:

1 million total vulnerability assessments

25,000 experienced experts in web application VA

$2,500,000,000 (US) in salary for web application experts

$5,000,000,000 (US) retail assessment cost

Even though the assumptions were way the conservative side, it’s immediately apparent that this scenario is completely fictitious. There are probably only 3,000 experts (a guess) in the world qualified to perform assessments relative to the 25,000 required. And much as I’d wish they would, enterprises are simply not going to spend multi-billions on web application security in 2007.

Of course as the awareness of web application security builds the numbers will climb, but for now we have to face facts. And the fact is unless we can vastly improve the web application VA process, most websites will not be assessed for security and remain insecure. That’s what’s going on today. And that’s why I’m saying the future of web application vulnerability assessment is about scale.

While we certainly can’t reduce the number of “important” websites, can reduce the number of man-hours and expertise required to perform an assessment using technology and a modern processes. Modern assessment processes need to be highly streamlined, repeatable, thousands running concurrently and performable by less than top-tier webappsec experts. This is what it truly means to “scale”.

How much improve can be made near term is a subject of much debate, but we’re working on it. For fun, let’s try a few more guesses at how certain efficiencies will help.

Future improvements:

500,000 “important” websites (roughly 1/2 of 1% of the total population)

Assessments 2-times a year per website. (Vary on change rate)

An expert can perform 40200 assessments per year with base salary of $100,000$80,000 (US).

Retail cost per assessment $5,000$2,000 (US).

Adjusted requirements:

1 million total vulnerability assessments

5,000 experienced experts in web application VA

$2,000,000,000 (US) in salary for web application experts

$400,000,000 (US) retail assessment cost

These numbers are much more palatable in the grand scheme of things and gives us our benchmarks for where technology and process must bring us to. How long will it take to get there is anyone's guess.

8 comments:

And to top it all off, the most serious of the vulnerabilities are impossible for any automated tool to find.

Source code scanning earlier in the development process and more reliable VA scanners will help, but the lions share of my VA time is spent working on logic flaws that neither a source code scan nor an automated application scan can find - horizontal and vertical privilege escalation (some will tell you their tools can, but they're not that great), XSRF, and privacy issues.

What will help to a degree is for coders to be more focused on coding correctly (no, we don't have to train them to use MITM proxies against their sites in order to do this) and for engineers to understand that security needs to be baked in from the beginning.

We're probably in a losing proposition, though. The bad guys always have an unlimited supply of able workers and the reward scale is such that money isn't necessary (initially) to motivate them.

I don't understand your math. You approximately halved the retail cost of an assessment, but the price went down an order of magnitude. I think the $2,000/assessment retail is incorrect, given that worker productivity went up over 50-fold. Did you mean to type in $200?

From nits to useful comments, I think that a lot of the reason that network assessments are cheap now has to do with the very things that make people say that web assessments are hard. Networks tend to be standardized, and net admins are educated about security. The parallels to this in the web app world are frameworks and education. Un/fortunately, I don't see either of these being at the maturity level of networks anytime soon. Network architecture has essentially stagnated. Nobody's deploying any radical new network architectures. Wireless would be the exception, and we can see how secure that is. I don't think web assessments will be as straightforward as net assessments until web development stagnates and frameworks emerge that are as widespread and secure as TCP or SSL implementations are today. With all the churn and parallel implementation that exists in web development today, I don't see that happening anytime soon.

(P.S. I get like a 50% success rate tops on your captcha. Does that mean I'm a replicant?)

> I don't understand your math. You approximately halved the retail cost of an assessment, but the price went down an order of magnitude. I think the $2,000/assessment retail is incorrect, given that worker productivity went up over 50-fold. Did you mean to type in $200?

In my model worker productivity increase 5-fold (from 40 assessments to 200) which could have meant an retail price decrease from $5,000 to $1,000. However, I don't think the two metrics are necessarily directly related. $1,000 was too cheap. I just picked some numbers that felt right based upon the trajectory I see at WH.

> Networks tend to be standardized, and net admins are educated about security. The parallels to this in the web app world are frameworks and education. Un/fortunately, I don't see either of these being at the maturity level of networks anytime soon.

That's a very compelling observation. That could very well be exactly what's going on. And frankly, a topic article worthy if expounded upon. I'll have to think about this more.

> (P.S. I get like a 50% success rate tops on your captcha. Does that mean I'm a replicant?)

HEHEH, I've had the same problem and wonder the same thing often about myself.

Oh, wait, I see the math problem. You swapped the assessor cost and the retail assessment cost in the "brave new world" scenario. It still costs billions of dollars to do the tests, but at least you can do it with only 5,000 people.

About Me

Jeremiah Grossman's career spans nearly 20 years and has lived a literal lifetime in computer security to become one of the industry's biggest names. He has received a number of industry awards, been publicly thanked by Microsoft, Mozilla, Google, Facebook, and many others for his security research. Jeremiah has written hundreds of articles and white papers. As an industry veteran, he has been featured in hundreds of media outlets around the world. Jeremiah has been a guest speaker on six continents at hundreds of events including many top universities. All of this was after Jeremiah served as an information security officer at Yahoo!