Computer product testing, sadly, has been as much art as science over the
years. It's not just that the products are so complicated as to defy simple,
straightforward analysis, but also there are no general agreements on how
products should be tested. Now that may be changing with respect to the testing
of anti-malware products.
New guidelines issued by AMTSO (Anti-Malware
Testing Standards Organization) set an excellent standard for high-quality
testing that you can believe in. I was in the professional testing business for
many years, at least 13 or 14, and was technical director at four different
labs. I don't do much actual testing of products anymore, but I still follow
testing issues carefully. I'm really impressed with what I'm reading in these
standards.

The effectiveness and performance of anti-malware products must be measured in a balanced way.

Testers must take reasonable care to validate whether test samples or test cases have been accurately classified as malicious, innocent or invalid.

Testing methodology must be consistent with the testing purpose.

The conclusions of a test must be based on the test results.

Test results should be statistically valid.

Vendors, testers and publishers must have an active contact point for testing-related correspondence.

Some of these are more obvious than others, but the
elaboration of the principles that follows makes clear they aren't just lip
service. With respect to No. 1, I've been involved with malware tests,
especially for the ability to detect unknown malware, where we have discussed
creating new malware purely for the test. The guidelines specifically forbid
this, although it does allow the modification of existing malware
characteristics. This principle also speaks about taking precautions to prevent
malware from escaping the lab.

Larry Seltzer has been writing software for and English about computers ever since,much to his own amazement,he graduated from the University of Pennsylvania in 1983.

He was one of the authors of NPL and NPL-R, fourth-generation languages for microcomputers by the now-defunct DeskTop Software Corporation. (Larry is sad to find absolutely no hits on any of these +products on Google.) His work at Desktop Software included programming the UCSD p-System, a virtual machine-based operating system with portable binaries that pre-dated Java by more than 10 years.

For several years, he wrote corporate software for Mathematica Policy Research (they're still in business!) and Chase Econometrics (not so lucky) before being forcibly thrown into the consulting market. He bummed around the Philadelphia consulting and contract-programming scenes for a year or two before taking a job at NSTL (National Software Testing Labs) developing product tests and managing contract testing for the computer industry, governments and publication.

In 1991 Larry moved to Massachusetts to become Technical Director of PC Week Labs (now eWeek Labs). He moved within Ziff Davis to New York in 1994 to run testing at Windows Sources. In 1995, he became Technical Director for Internet product testing at PC Magazine and stayed there till 1998.

Since then, he has been writing for numerous other publications, including Fortune Small Business, Windows 2000 Magazine (now Windows and .NET Magazine), ZDNet and Sam Whitmore's Media Survey.