Some people eagerly await publication of a new thriller from a favorite author. As for me, I get that thrill when one of the antivirus test labs releases a new report. Will there be any upsets? Any new products in the mix? The latest test from Dennis Technology Labs certainly offered a surprise twist.

Who They Test In each quarterly test, researchers at Dennis Labs evaluate eight popular antivirus products, two of them free. They also include Microsoft Security Essentials as a baseline. A product that can't do better than the free protection built in to Windows, well, that product needs work.

Recently, they've added something new. Each quarter features a "guest antivirus," a product that will go through the full testing regimen but that won't be added to the continuing rotation.

How They Test To measure a product's effectiveness at protecting users from real-world attacks, the researchers locate actual malicious websites and record all of the Web traffic involved in the attack. A playback system lets them subject each antivirus product to the exact same attack.

They note whether the product completely blocked the attack (three points), caught the malware after installation but completely wiped it out (two points), or neutralized the malware without actually removing its traces (one point). If a product doesn't meet any of these protection levels, they launch an on-demand scan to see if it will detect the malware, potentially earning one or two points. An antivirus that allows the malware free rein on the test system loses five points. Over the course of the quarter they run 100 such tests, always using the very latest malware.

A separate test replaying downloads from valid URLs tests the antivirus tool's accuracy from the other direction. Naturally a valid download shouldn't be blocked; products lose points for these false positives. The false positive scores are weighted based on the valid program's prevalence.

The Mystery Guest During the last quarter, the rotating guest antivirus spot was occupied by the free Malwarebytes Anti-Malware 2.0. This product is a bit of an odd man out, given that it doesn't include any malware-blocking capabilities. However, I expected it to succeed at the on-demand scan phase. My expectation was entirely wrong, it turns out. Malwarebytes neutralized just a tad over 60 percent of the samples.

With no real-time blocking, Malwarebytes couldn't possibly earn any three point scores. If it fully remediated all of the attacks, its maximum possible score would have been 200 points out of a possible 300. But it didn't. In fact, it scored negative 67 points. Only Microsoft came in lower, with negative 89 points.

It does seem a little unfair to handicap Malwarebytes with the impossibility of achieving any full three point scoress. What if we gave it three points for complete remediation, given that's the best result it can achieve? I ran the numbers. Even with this modification, the score still comes in below zero.

Top Products Dennis Labs issues five levels of certification, AAA, AA, A, B, and C. Kaspersky, Norton, and ESET hit the AAA level. Malwarebytes didn't manage certification at all. You can read the full report here.

Malwarebytes has a good reputation as the go-to solution when other antivirus products fail. In the past, it has scored well in my hands-on malware cleanup tests, and it's our current Editors' Choice for free, cleanup-only antivirus. This result is pretty awkward. The only resolution I can see is for Malwarebytes to participate in testing by more of the independent labs, so this isn't the only mark on their scorecard.

About the Author

Neil Rubenking served as vice president and president of the San Francisco PC User Group for three years when the IBM PC was brand new. He was present at the formation of the Association of Shareware Professionals, and served on its board of directors. In 1986, PC Magazine brought Neil on board to handle the torrent of Turbo Pascal tips submitted b... See Full Bio

Get Our Best Stories!

This newsletter may contain advertising, deals, or affiliate links. Subscribing to a newsletter indicates your consent to our Terms of Use and Privacy Policy. You may unsubscribe from the newsletters at any time.