The AMTSO (Anti-Malware Testing Standards Organization) meeting in Prague, which took place at the beginning of this week, proved to be rather more exciting than you might expect from a group with the word "Standards" in its name.

One of the issues that caused particularly lively debate centred around the question of what constitutes AMTSO compliance, whatever you might understand by that term. When a tester claims to be AMTSO compliant – and many have started to do that – or uses phrasing that implies compliance, such as "following AMTSO principles", what does that mean?

Well, up to now, such phrasing has meant less than you might think it does, because AMTSO hasn’t formally defined what "AMTSO compliance" actually is. This has led to a certain amount of confusion, not only as regards tester’s claims to be compliant, but because AMTSO’s stance has been misinterpreted as meaning that dynamic testing is automatically compliant, while static testing is automatically non-compliant. (I don’t think this is at all the case, but I’ll come back to the static versus dynamic versus hybrid testing topic another time.)

What concerns me right now is that bitter experience suggests that if a tester makes a point of claiming that his methodology is conformant with the AMTSO guidelines, quite a few people will accept that claim uncritically.It seems to me that there’s a need for AMTSO to take ownership of the term "AMTSO compliant" before someone else (or, even worse, everyone else) does. In fact, some recent events have forced the organization to start thinking about specific steps in that direction. While nothing is finalized, it’s likely that in order to minimize the possibility of abuse and a definitional free-for-all, these steps will be based on the idea of self-assessment that the organization was already considering.

This doesn’t, of course, mean that anyone is going to be able to say "Yes, of course we’re compliant." Rather, I’d envisage that testers wishing to use the term or something similar will have to complete a self-assessment form, which will have been received and acknowledged by AMTSO, and make them accountable to AMTSO for the use (or misuse) of claims of compliance.

In the meantime, I’d strongly recommend that if you come across claims of "compliance", you take them to be as a declaration of intent to comply: it doesn’t mean that they are proven to comply or have the blessing of AMTSO.

I’d guess (or hope) that eventually you’ll be able to check on the AMTSO web site as to whether a given tester has completed the self-assessment process (when it actually exists). Even then, since AMTSO is not a certification body (not yet, anyway – who knows what will happen further down the line?), it probably won’t mean that any specific test from that tester or organization is compliant. Unless, of course, an analysis from the Review Analysis Board has determined that it is.

Even if the tester is a member of AMTSO, that doesn’t mean at all that they have the automatic endorsement of the organization for their testing. Indeed, they’re at least as liable as anyone else to have their adherence to the AMTSO principles scrutinized by the Review Analysis Board.