Which Antivirus Cleans Best?

Many antivirus lab tests measure the percentage of samples that antivirus products can detect. An interesting test from AV-Comparatives takes detection as a given and measures just how well antivirus products clean up the malware they've detected. This year's scores overall are better than last year's

When you install antivirus protection to clean up a malware-infested PC, two things should happen. First, the antivirus should detect the active malware; second, it should thoroughly remove the infestation and remediate its effects. The latest test from AV-Comparatives specifically evaluates how well an antivirus manages the cleanup process. Overall, the results look better than the introductory run of this test in 2011.

Products Tested If you look at both tests, you'll notice the list of products tested is rather different. This time around AV-Comparatives only included products whose vendors subscribe for testing at the Premium level. K7, McAfee, Microsoft, Qihoo, Sophos, Trend Micro, and Webroot were tested last year, but as they subscribe at the Basic level they didn't make the cut this time. BullGuard, Fortinet, and GFI Vipre are new additions, not tested last year. This year's test looked at 13 products in all.

Symantec's researchers have a beef with testing methods in one particular AV-Comparatives test. AV-Comparatives won't let vendors pick and choose which tests they'll participate in, so this year Symantec isn't participating at all.

Test MethodologyBecause this is very specifically a test of malware removal, the researchers only chose malware samples known to be detected by all of the tested products. They further ensured that each sample is found in the wild, and that it doesn't perform irreversible destructive actions. Finally they narrowed down the possibilities to 14 samples that demonstrate common malware behaviors.

The researchers thoroughly analyzed each sample, recording every change made to the infested system. For each product, they installed the sample on a clean test system, rebooted, and verified that the sample was active. Next they installed and updated the antivirus and ran a full scan. After rebooting, they checked the test system manually to see how well the antivirus product cleaned up.

Rating Convenience When you install antivirus protection to clean up a threat, you want to just run a scan and be done with it. A product that completed the cleanup process in normal Windows earned an A for convenience. If installation or cleanup required rebooting in Safe Mode or running a separate utility, that product got a B. Most of the tested products offer a Rescue Disk for tough cases. Resorting to the Rescue Disk isn't very convenient, so products that required it get a C. Finally, if the cleanup utterly failed or couldn't be completed without intervention by tech support, that's worth a D.

Rating ThoroughnessAfter each product did its best to clean up each sample, the researchers dug in and manually checked to see how good a job it did. To earn an A for cleanup, the product had to remove everything except negligible, non-executable traces. If the product did disable the sample but left behind significant traces like executable files or changes to the MBR, that earned a B. A product that removed the malware but left annoying or dangerous problems rated a C. Examples of such problems include disabled task manager, disabled registry editor, and compromised HOSTS file. If removal failed or the system became unusable, the product got a D.

To get the final score for a given combination of antivirus and malware sample, the researchers used a simple scoring system in which A for removal and A for convenience is worth 100, A for removal and B for convenience is 90, and so on down to 0 points for D and D. A product's final score is simply the average across all fourteen samples.

Even the lowest scores weren't dreadful. AVG Anti-Virus FREE 2013 earned a B average for both thorough removal and convenience, while avast! Free Antivirus 7 averaged a B for convenience and B- for thorough removal. These got a STANDARD rating, the lowest passing rating. All of the others were rated ADVANCED.

The full report goes into significant detail describing the malware samples used and lists the precise score for each product with each sample.

It also lists a number of ways that antivirus vendors could improve their scores, and thereby improve their product's ability to rid their customers of malware. For example, vendors should provide an offline installer in case malware interferes with a connection to the vendor's website. Antivirus installers should check for and deal with active malware before attempting installation. The antivirus tool itself should point to standalone tools if installation or cleanup fails. Vendors, take note; these are really good ideas!

Neil Rubenking served as vice president and president of the San Francisco PC User Group for three years when the IBM PC was brand new. He was present at the formation of the Association of Shareware Professionals, and served on its board of directors. In 1986, PC Magazine brought Neil on board to handle the torrent of Turbo Pascal tips submitted by readers. By 1990, he had become PC Magazine's technical editor, and a coast-to-coast telecommuter. His "User to User" column supplied readers with tips...
More »