More Blogs

Webroot Accused, Exonerated

A week or so ago I received a disturbing link from several sources. It pointed to a review in which Webroot SecureAnywhere Antivirus scored below almost all of its competition. That low score isn't what bothered me. It's not uncommon that the same product scores differently under different test methodologies. The kicker was an anonymous comment claiming that Webroot cheated in PCMag's tests and that the company has pulled out of third-party tests where cheating isn't possible.

The commenter, using the name "the cloud," stated, "We had a team dedicated for 2 weeks to game Neil Rubenking's PCMag test for Editors Choice… Neil let us run multiple scans and after we gathered all of the malware files… our team hand selected all of the files to be detected on the next scan." Clearly we're intended to believe this comes from a Webroot employee. He went on to say, "This is also why you see Webroot pulling out of detection tests run by AV testing organizations."

In my review, I said, "Any time [Webroot] finds a threat, it asks permission to run a quick scan to find any remaining traces... Each time the 'remaining traces' scan found a new threat, it requested another scan… By the time Webroot finished, every single sample had been eliminated. That's a first!" According to the nameless commenter, what I saw reflected was not thorough cleanup but plain old cheating.

Not ImpossibleThe nastiest part about this accusation is that the scenario it describes is not impossible. My testing is a small operation compared to the dedicated test labs. For example, where NSS Labs conducts tests from 37 different countries using rotating IP addresses, my own tests always come from the same computer and IP address. In most cases I install the antivirus software using a registration key supplied by the vendor. There's plenty of opportunity to identify my testbed and give it special handling.

The thing is, nobody has ever tried this. And unless they could do the same to all the independent labs, it really wouldn't make sense. The chicanery would be revealed eventually, and the company's reputation would plummet. Wouldn't it?

Other TestsWebroot is definitely not pulling out of third-party tests. That portion of the accusation was just plain wrong. Only this week AV-Comparatives released not one but two test reports that include Webroot's antivirus. A brand-new malware removal test specifically analyzed how thoroughly 18 popular antivirus tools could remove existing threats detected by all of them. Another test measured the performance drag imposed by 20 antivirus tools. Webroot rated ADVANCED, the second-highest rating, in both tests.

Webroot will appear in the next antivirus certification report from AV-Test.org. The testing has finished, but they're still working on boiling down the reams of data. I got a peek at preliminary results, which suggest a very good score for protection against attack and a good (but not quite as good) score for malware removal. Full results for usability weren't in, but Webroot displayed an unusually high number of false positives (good files identified as malicious).

Virus Bulletin is also currently collating test results for the Webroot antivirus. The full report should be released soon, probably next week. It will show that detection results weren't great, and that the product threw an unprecedented number of false positives.

NSS Labs typically tests business security software, not consumer products. Where most independent labs are paid by the vendors to run tests, customers of NSS Labs are big companies looking to choose the right security products. I'm working with NSS Labs on the possibility of including Webroot in their next round of testing.

As for the review that started all this, it was written based on data provided by UK-based Dennis Technology Labs, but the review and interpretation are not from Dennis. The accusatory comment came up for discussion in theWilders Security Forums; one long-time member ran his own informal test and concluded that Webroot's repeated mini-scans to fully clean up found threats really do work.

Webroot ExplainsOne niggling problem remains. The behavior I reported, in which Webroot made repeated small scans and detected more threats each time until all were gone, is consistent with the accusation that technicians at Webroot's end were actively gaming my test. If not that, what really happened?

I spoke with Mike Malloy, Executive VP, Products and Strategy, and with Joe Jaroch, Chief Security Architect, to get a full understanding. Joe is responsible for the from-scratch rewrite that made Webroot by far the smallest antivirus solution around. What other antivirus could fit on a 3.5" diskette… twice?

They explained that when Webroot detects a high concentration of threats it goes into a kind of alert mode, tightening up its behavioral detection rules. It also alerts Webroot's researchers. I should have remembered this; they told me the same thing when we met in August.

This feature is designed to handle systems that are badly infested by multiple threats. It does also have the effect of cranking up protection during some kinds of testing. Jaroch stated that while Webroot identifies locations like the Desktop, it does not send full pathnames and in particular does not send the username. He went on to note that Webroot guarantees malware removal, including remote-controlled manual removal by a technician at no extra charge.

I have some ideas for future tests that may avoid triggering the alert mode, just to see what happens without it. But I'm satisfied that my existing test wasn't gamed.

Why, then, would this nameless netizen effectively accuse me of being a gullible moron and Webroot of cheating? My Webroot contacts couldn't answer that, naturally. They did point out that the development staff experienced some turnover when they discarded the old code base. Conceivably a disgruntled former employee could be stirring up trouble. We'll never know unless "the cloud" steps forward and identifies himself.

Automatic Renewal Program: Your subscription will continue without interruption for as long as you wish, unless
you instruct us otherwise. Your subscription will automatically renew at the end of the term unless you authorize
cancellation. Each year, you'll receive a notice and you authorize that your credit/debit card will be charged the
annual subscription rate(s). You may cancel at any time during your subscription and receive a full refund on all
unsent issues. If your credit/debit card or other billing method can not be charged, we will bill you directly instead. Contact Customer Service