Test environment

VBSpam is a twice-monthly comparative test for spam filters.

Products must be submitted in a form in which they are available to the end-user.
Developers will be permitted to make changes (should they be required) in order for their product to
function in the testing environment - but in all instances the VB test team must be made fully aware
of such changes.

It is the
developers' responsibility to inform the VB test team of any set-up or configuration changes that need to be made in order for the
product to work in the testing environment.

Each product will receive all emails from the same fixed IP address; filtering that is based on the connecting IP address
thus cannot be used. However, products may perform IP or domain-based filtering using the information
in the Received headers that will be added to the emails to reflect the fact that the
email has passed through the test network's MTA. If requested by developers, the IP address can also be added to
the emails in an X-Forwarded-For header. There is also the possibility of providing a filter with this information during the SMTP transaction, thus enabling pre-DATA filtering.

As every email is relayed from a fixed IP address, greylisting should be turned off. If an
email cannot be delivered, up to five redelivery attempts will be made with 20-minute intervals, but this can be
disabled on a per-product basis if required. However, unlike in previous tests this does not include 5xx responses during the SMTP connection; these are considered to mark the email as spam and emails will not be sent.

Each product, as far as it contains or works together with an MTA, is required to relay the filtered emails to a
back-end MTA; the classification should be mentioned in the header in a clear and unambiguous way. Email that has not reached the back-end
MTA one hour after its original delivery will be considered to have been marked as spam.

The VB test team should be provided with a technical contact with both a good knowledge of the product and an understanding of the testing environment.

Classifying

All products are required to classify each email into one of two categories: 'ham' or 'spam'. If a filter uses other
classifications (e.g. 'phishing', 'virus', 'possible spam'), the test team must be informed as to whether these
classifications are to be considered ham or spam.

Products are not required to check emails for malware, but spam emails containing malware
should be marked as spam. Should a ham message contain a malicious attachment, it will be
removed from the test set (thus a product will not be penalized for blocking malware).

Filters will not be trained.

Filters are not permitted to use any ad hoc rules based on properties of the mail streams used in our test, e.g.
automatically blocking all email in certain languages or alphabets, or whitelisting certain senders. The testers will perform regular tests to make sure this is not happening.

Email corpus

The email corpus consists of the traffic sent to a number of legitimate mailing lists, as well as spam corpuses provided by Project Honey Pot and Abusix, which are randomly
assigned to an address on the vbspamtest.com domain. All emails are forwarded in real time. Most of the mailing list emails have their headers rewritten so that they appear to have been sent from the original sender directly to Virus Bulletin; more on which can be found here.

The products will receive approximately one email every two seconds.

Each email will be left unchanged with the exception that of a Received header added as follows:

where the date is the current date and time, 12.34.56.78 is the IP address of the
sending MTA, yyy.yyy.yyy is the domain in the HELO/EHLO command, and
xxx.xxx.xxx is its reverse DNS record (defaults to Unknown if no reverse
DNS record is found). The string dddddddddddddddddddd.ppp is used by the back-end MTA to uniquely identify both the email and the product.

The HELO/EHLO domain of the SMTP envelope will be changed to reflect delivery from a local MTA; the
MAIL FROM and RCPT TO values will be the same as those of the original message.

Awards

On completion of the test, a weighted false positive (WFP) rate and average spam catch
(SC) rate (#spam caught/#spam) will be computed for each product.

The WFP rate is defined as the false positive rate of the ham and newsletter corpora taken together, with emails from the latter corpus having a weight of 0.2:

Products whose spam catch rate minus five times the weighted false positive rate (SC - [5xWFP]) is greater than or equal to 98 will earn a VBSpam award:

(SC - [5xWFP]) ≥ 98

Meanwhile, the VBSpam+ logo is awarded to products that combine a spam catch rate of 99.50% or higher with zero false positives and no more than 2.5% false positives among the newsletters.

For each product, up to four false positives will be counted per sender, where two senders are considered to be the same if the value of MAIL FROM is the same or very similar and the sending IP address is on the same /24 network.

The comparative review will be available free of charge
and does not require registration or subscription.

Once a product has been accepted for testing, it may not be withdrawn from the review without 21 days notice from
the vendor - full details are provided in the vendor test agreement (please contact us for a copy of the test agreement).

This site uses cookies to store information on your computer.
To find out more about our cookies and how to manage them, take a look at our Cookie Information page. By using our site, you agree to our use of cookies and our terms and conditions.