Categories

How Good Are Website-Reputation Services?

Websites on the Internet have now become the standard modus operandi for spreading malicious software to infect personal and corporate environments. A large number of benign and well-meaning websites are compromised everyday by hackers inserting malicious code to, in turn, infect the computers used by visitors to the hacked site. One of the ways to combat this is to develop a website reputation mechanism which can warn of potential threats before visiting a compromised site.

Website-reputation services vary wildly in their opinions.

Note that all 350 domains, were reported as malicious, and were collected from malware.com.br on December 18, 2009. The blue column (maximum 350) indicates the number of sites that the website-reputation service correctly identified reported bad sites. The orange column (maximum 350) indicates the number of sites that the website-reputation services incorrectly identified reported malicious sites as safe.

Website reputation services have been around for nearly 5-7 years now. Initially developing as a niche product line which could serve to provide an opinion of a site’s reputation to full fledged offerings which provide advisories about websites, whether they are distributing malware, and if they are, what kind, and using which Autonomous Systems.

At StopTheHacker.com (Jaal LLC) we have conducted tests with 350 domain names, all of which have been reported as malicious by volunteers of various blacklists.

The aim of the test is to:

Identify how accurate the website reputation services are

What is the overlap in terms of safe/unsafe websites

We have found some interesting results which we present in this article. First we detail the parameters of the testing procedure to provide an idea of how the test was set up.

350 URLs were collected from malware.com.br (mbr) on December 18, 2009. These URLs are reported to this website for listing by one or more of the following: individuals, organizations, agencies and software products or services. We assume for the purposes of this test that all the URLs obtained from the “regular” list from mbr are malicious and hence deemed “unsafe” to visit.

We compare the reputation provided by each website-reputation service and observe how many websites are marked as unsafe, safe, untested, maybe-unsafe/caution/potentially-unsafe, unreachable.

Note, that when analyzing a domain name, for checking with the Google safebrowsing API, we have had to calculate the MD5 hashes of the website names to match with the malware hash list. The date that we conducted this test was: December 21, 2009. The list of domain names tested are presented below and a graph representing the statistics for the first 350 sites tested is above.

We have identified some of the most interesting results below:

McAfee Siteadvisor marked 32.5% of Domains as Unsafe, 22% as Safe, 43% as Untested and 1.7% as Potentially-unsafe.

Norton Safeweb marked 50.86% of Domains as Unsafe, 43.71% as Safe, 2.29% as Untested and 3.14% as Potentially-unsafe.

Google SafeBrowsing marked 10.86% of Domains as Unsafe, 89.14% as Safe. Note: the presence of the hash of the domain name being tested, on the google malware hash list, is interpreted as “unsafe” while the absence in interpreted as “safe”.

Comodo Siteinspector marked 0.29% of Domains as Unsafe, 98.86% as Safe and 0.86% as Unreachable. Note: after feedback from Comodo, a retest was conducted, accuracy changed from 0.29% -> 1.2%.

This limited test is a first step towards showing how much variance there is website reputation services that are currently being offered by large Internet-services/security companies. To highlight this point we present immediately below the relatively few domains (~6% of the total domains tested) that were marked as bad by all three major services, Norton, McAfee, and Google.

In brief:

6% of domains tested were marked as “unsafe” by all 3, McAfee, Norton and Google

10% of domains tested were marked as “unsafe” by Norton and Google

22% of domains tested were marked as “unsafe” by Norton and McAfee

5.7% of domains tested were marked as “unsafe” by Google and McAfee

Update: December 28, 2009

After receiving helpful feedback from representatives at Comodo, we were informed that Comodo’s service could provide more accurate answers if complete web page locations were checked instead of just the domain name. We followed the advice and saw a definite increase in Comodo’s accuracy. Comodo marked 1.2% of the website/pages as malicious. Prior to this re-test, the same service marked 0.2% of the websites as unsafe. The graph at the beginning of this article does not represent the results of this re-test.

Interestingly, Comodo’s service marked only 1 website, 218.146.255.156 as malicious. This domain was also marked malicious by Norton, “Untested” by McAfee and was not found on the Google malware hash list. Below follows the complete list of domains that were tested.