> On Thu, Jan 21, 2010 at 1:53 AM, Michal Zalewski <lcamtuf@coredump.cx>> wrote:> >> Testing takes time. That's why both Microsoft and Mozilla test.> >> > Testing almost never legitimately takes months or years, unless the> > process is severely broken; contrary to the popular claims,> > personally, I have serious doubts that QA is a major bottleneck when> > it comes to security response - certainly not as often as portrayed.>> There are a lot of factors that go into how long it takes to run QA.> Here's a few (I'll leave out the joys of multivendor for now):>> 1) How widespread is the deployment? A little while ago, Google had> an XSS on Google Maps. An hour later, they didn't. About a decade> ago, AOL Instant Messenger had a remote code execution vulnerability.> Eight hours later, they didn't. Say what you will about> centralization, but it *really* makes it easier and safer to deploy a> fix, because the environment is totally known, and you have only one> environment. There are a couple dimensions at play here:>> a) How many versions do you need to patch?> b) How many different deployment classes are there? If your> developers are making a bunch of enterprise assumptions (there's a> domain, there's group policy, there's an IT guy, etc) and the fix is> going to Grandma, let me tell you, something's not going to work> c) What's at stake? Your D-Link router has very different deployment> characteristics than your Juniper router.>> 2) How complicated is the fix? Throwing in a one-liner to make sure> an integer doesn't overflow is indeed relatively straightforward. But> imagine an oldschool application drenched in strcpy, where you've lost> context of the length of that buffer five functions ago. Or imagine> the modern browser bug, where you're going up against an attacker who> *by design* has a Turing complete capability to manipulate your object> tree, complete with control over time. Or, worst of all, take a> design flaw like Marsh Ray's TLS renegotiation bug. People are still> fiddling around with figuring out how to fix that bug right, months> later. Complexity introduces three issues:>> a) You have to fix the entire flaw, and related flaws. We've all seen> companies who deploy fixes like "if this argument contains alert(1),> abort". Yeah, that's not enough.> b) You have to not introduce new flaws with your fix -- complexity> doesn't stop increasing vulnerability just because you're doing a> security fix.> c) The system needs to work entirely the same after. That means you> don't get to significantly degrade performance, usability,> reliability, or especially compatibility. Particularly with design> bugs, other systems grow to depend on their presence. No software> lives in a vacuum, so you have to actually _find_ these other pieces> of code, and make sure things still work.>> 3) How many people do you actually expect to deploy your patch?> There's this entire continuum between "only the other developers on> SVN", through "the people who call to complain", to "everybody who> clicks 'I accept the risk of patching'", to "my entire customer base> with zero user interaction whatsoever". A patch with problems 0.005%> of the time is acceptable if 1000 people are deploying, but not if> 1,000,000 people are deploying. Note that security research is very> strongly correlated with deployment numbers, to the point that> vulnerability count is much more correlated with popularity than code> quality. So you have this interesting situation where the more your> fix is pushed, the more scrutiny there will be upon it.>> Now, you can consider these all excuses. Believe me, QA people have> no shortage of guys who look down on them and their problems. But> certainly different bugs have different characteristics, and assuming> that all things can be fixed in the same time window with the same> output quality is just factually untrue. You might as well be> claiming the next version of HTML5 will include an <antigravity> tag> that will make your laptop float in the air and spin around.>> There is a balancing act. Years is, of course, ridiculous. In many> situations, so too are a couple of weeks. If the goal is to achieve> the best quality patches, then you want the issue _prioritized> heavily_, but not _on public fire_. The latter encourages people to> skip testing, and you know of course what happens when you skip> testing?>> You end up with Gigabit Ethernet drivers that can't actually handle> all frame lengths. (Epic Intel find from Fabian Yamaguchi. Wow.)>> Responsible disclosure has its risks. You really can be jerked around> by a company, *especially* one that hasn't experienced the weirdness> of an external security disclosure before. But engineering realities> don't go away just because the suits finally got out of the way.> Testing matters.>> _______________________________________________> Full-Disclosure - We believe in it.> Charter: http://lists.grok.org.uk/full-disclosure-charter.html> Hosted and sponsored by Secunia - http://secunia.com/>