May 10, 2008

What makes a Security Project?

Why is it that when you come across a good new thought, it is harder to deal with than an old, rehashed thought? I struggle with this all the time: E.g., blogs. my favourite ones are the writers that do original and new thinking. These guys nibble and munch at problems until they find answers. Then they bake solutions. These posts are so full of good stuff that I don't know where or how to respond. On the other hand, my unfavourite blogs are the ones that stick very clearly in the middle ground, express mildly polemic thoughts that a majority agree with and a minority already said, and seem to spend more time collecting and building popular support than anything useful.

Lots of good posts these days over at Gunnar's area, and I can't easily respond to them.

I see no evidence that [Sun] understand the need for writing secure code more so than Microsoft. In fact I see every evidence that Sun is several years behind Microsoft on software security. Let's do the list - Howard/Leblanc's work, threat modeling, software security patterns and practices, SDL, SecPal, BlueHat, OWASP guidance work and that is all before we get to identity stuff.

You won't see such an ... *opinion* from the popular fence sitters! Why is this? I think it is for several reasons. To say such a thing means you court disfavour with large companies, including the one you named, but also other companies who might realise you are likely to bark with more bite than other tame consultants.

Further, one has to think of the evidence to back up the opinion, and that's not always easy. I know because I tried to clarify this three years ago, while dealing with the question. When I sat and thought about why I thought some organisations weren't up to scratch, I had no easy answers. So I wrote down everything I could think of ... and then judged every organisation I knew on my list of metrics.

For once, then, I can respond to Gunnar, and in full wide-screen TV mode:

To make sense of that, you will have to check out the fuller essay. Even then, note that it was never finished, and the opinions are already 3 years old. As to whether Gunnar is right, check the table metrics, calculate your view and decide for yourself!

Do you want commentary on the general scoring criteria, or me to point out errors in your results?

There is pretty much zero, let me repeat, zero evidence that open source products are more secure. More transparent perhaps, but not more security. Including it as a useful metric for judging the security of a product is more than a little ridiculous given what we know current about public defect rates in open vs. closed source products.

As much as I like the work the OpenBSD community has done, their stance on certain classes of vulnerabilities has been pretty weak/lame to date. Consider last year's kernel bug that the core impact guys found. At first since it was "only a DoS attack" they refused to categorize it as a security bug. Well, if an attacker shuts most people down they are going to consider it a big deal.... oh, and it turns out it was exploitable to get root... hmm....

But, enough about methodology..... how about the data.

As much as I've never been a Microsoft apologist in my past, they are actually leading the pack in this area. Perhaps not in terms of absolute defect rate (they started with a bit of a self-imposed handicap there) but in terms of current SDL methodology and openness. Check out their SDL blog, Michael Howard's blog, and the MSRC blog for exhaustive details on a number of recent security defects. Of commercial organizations they are being more open than anyone about the causes of defects and how to counter them.

You've said that the chart is a bit dated, perhaps we could take a look where things stand in 2008 and get a better picture?

The Linux kernel has a broad set of crypto goodies built into it, of varying quality. It randomness source is of questionable quality, its disc encryption modules are very good, its password encryption stuff is standard, high-quality, etc.

Also, I strongly disagree with Andy. The defect rate is a very poor metric, IMHO. Not all defects are equal. You cannot just add them up.

To answer Andi's questions on scoring: Criteria is more important, I think. As the scoring is three years old, I reckon pointing out errors is futile. As you suggest, a lot has changed, and I agree that Microsoft has done a whole lot more than other organisations in that time.

Should we take a look at the different activities and update it to 2008? I don't know ... my question would be, how do we ensure some sort of relatively unbiased approach? How do we avoid mobbing by the apologists for the manufacturers?

I think the commentary on the criteria is what I'm more interested in, either developing them, expanding them or trashing them. There are some unusual artifacts in there that make one think that maybe the criteria need more thought.

and then they asked me to lead a whole software security track. This kind of thing happens all the time. I am not critiquing companies as a loose cannon, but earnestly hoping they do a better job. I realize its a new field. I criticize Sun in particular because I hold them to a *higher* standard than others.