Infosecurity Magazine on AMTSO's credibility gap

I was interviewed yesterday by Fred Donovan, following up on the paper on AMTSO I presented at EICAR earlier this month. I may be prejudiced, but I think he's summarized my current thoughts on the topic pretty well in the article, though it isn't my recommendation that the existing guidelines be reviewed independently: it was one of the suggestions that came out of the last workshops. Not that I'm against it, either: it might be one way of giving them more credibility, but I'm not sure it would transform them from guidelines to standards.

Whether AMTSO's new executive team will agree, is another question. I look forward to seeing how that initiative pans out.

But for myself, I continue to consider it essential for AMTSO – or an organization including or replacing it – to have better credibility than it does right now: if this initiative fails, testing is, in my eyes, close to useless because there will be no impartial authority to hold testers to account for the accuracy of their conclusions, and in the long run that will hurt their credibility. Hat tip to @imaguid for forcing me to crystallize that thought, unpalatable though it is.

There's a bit of a conundrum here. In many respects, the companies most qualified to evaluate test procedures are the companies themselves. They would also like to perform better in these tests, than their competitors. This is a motivation to find any flaws in the tests or analysis of the results that would put them at a disadvantage.
As part of this, and I believe the AMTSO has a white paper on it, has to do with sample size, and statistical significance. Many testers seem oblivious to this issue, which immediately harms the validiity of their analysis and conclusions. If one company misses 0.004% of a large sample, and another company 0.008%, for marketing purposes a tester/reviewer will write, "Twice as effective!". There are some stats on testing where results published on vendor's web sites also show the inability to perform simple division.