Faulty math

Recently White Hat Security released some “research” they have done into web applications they have audited with the purpose of identifying the “most secure programming language or development framework available”. The result, they believe, is all languages are equally insecure. I however feel that their “research” misses the point entirely and proves very little.

I want to be very specific here. This research may legitimately state: of the 1,700 sites we audited we found these xyz results. However, it’s not that which I have issues with. Instead it’s the extrapolation saying “all languages are equally insecure” I find faulty. Specifically there are three major problems I can see:

Narrowed Focus

The fact the matter is that the data they collected is from people who hired them to audit their sites in the first place. By doing so, the playing field of sites has already been narrowed down to a specific group of companies. I mean think bout it… people who feel sick go to hospitals. So, is it shocking that regardless of who they are and what symptoms they have… they often are?

We can’t reasonably say that: Of the 1000 people who came to our hospital, we have determined 900 of them to be sick. Therefore, 90% of all people (inside and outside the hospital) are ill. Math doesn’t work that way unless you are running for political office.

Ambiguity

The research seems to ignore the skill and processes that may be used in the development of said 1,700 websites. Some sites are built by larger companies than others, some are likely built by software firms, etc… This is incredibly relevant, as if the research is leveraging a mishmash of skill and size (which it is), then the results will be skewed. If it really did take companies days, weeks, and even months to fix vulnerabilities, it’s obvious you aren’t talking apples and oranges.

Think of it this way, if I had a baseball team with 10 hitters. 9 of them are little league players and can’t hit the ball 90% of the time, but one of them is a pro-league player and always hits the ball. On average you could fairly say: 81% of team strikes out. But, does that prove that on average 81% of ALL baseball players (regardless of league) can’t hit? Of course not.

It only proves that most little league hitters can barely hit the ball. They don’t have the time and experience on the plate yet to be consistent and reliable. The EXACT same thing is true in software engineering and security.

Premise

Grossman said in an interview that: “…one Web programming language / development framework can be made basically just as secure (or not) as any other”. While I agree with that to some degree, I think it is a radical over simplification.

Computer languages at their core are naturally inclined to solve a particular set of problems. This is, as a matter of historical fact, why we have so many. Therefore, languages in their focus to solve one subset of problems, make it both easier to do some things, and HARDER to do others. For instance, I would personally argue that securing a php website over an mvc.net one is exponentially more difficult. This is due to: The nature of the language itself, and the available frameworks to cover surface area quickly.

For instance, the new mvc.net 2.0 framework has made encoding inputs so painless that it’s an almost non-issue. Compare that to ensuring that every field manually is encoded in other frameworks! It even has the ability to plug in various encoder/decoder frameworks as a configuration option. It also has an input tampering module that is enabled by default.

Why? Simply put Microsoft made security a priority in this newest release and now it’s EASIER to do the right thing.

Conclusion

Making decisions with factual information is a good thing. Extrapolating “fact” out of that can be a sticky situation, however. In my opinion I think that reports like this don’t prove or disprove anything. These are good numbers to have, but it surely doesn’t say what people seem to think it does.

Personally for my money (since they are both free) I think that the work MS does showing the year after year decrease in vulnerabilities in their products is invaluable evidence. I’d rather see how a company solves problems then how a bunch of them on average don’t.

It’s my opinion (not fact) that security process, training and priority does MORE to protect security problems then anything else. Including WAFs.

Final final thought: If it really does take you months to fix security vulnerabilities in your products it’s clear you either don’t care enough to do anything about them, or you have really poor developers &| process.