Blacklists, whitelists, and greylists are all essential spam-fighting tools. Here's how your organization can use them effectively

Like the rising cost of postage stamps, increasing complexity in e-mail is inevitable. In the early, halcyon days of the Internet, SMTP connections flowed like a mountain spring and mail filters were used solely for mail organization. Now, the water is brackish, and mail filters are an absolute necessity.

But whose filters? Given the extraordinary volume of e-mail that most organizations receive, care and feeding of e-mail whitelists and blacklists is sporadic at best, and it’s usually done only to address an acute problem. Subscription services such as Postini can alleviate this problem from an inbound perspective, but that’s only half the battle.

Free DNS blacklists such as spamhaus.org and spamcop.net provide an interactive service to enable inbound mail servers to match the IP address of the server delivering mail against a list of known spamming servers via a simple DNS query. If a positive match is returned, the mail is rejected.

Many organizations also rely on whitelists, which are simply lists of domains, addresses, or SMTP relay IP addresses that are always allowed to deliver mail. In most infrastructures, this is a list of domains that are close partners with the company, and ancillary addresses or domains that would be caught in a spam filter but are valid.

The remaining list-based protection form is greylisting. A greylist rides the boundaries of the blacklists and whitelists, using interpretive back-end code and SMTP status flags to create dynamic whitelists and blacklists.

All three approaches have their place in the modern enterprise’s battle against unwanted e-mail, but as with many well-intentioned schemes, caution should be exerted to protect the innocent, particularly when it comes to blacklists.

The vigilante approachAlthough quite plentiful, DNS blacklists have had their share of controversy. Given enough subscribers, a listing on a DNS blacklist can render e-mail useless for the target. Of course, this is the whole idea, but it’s not uncommon to find a site listed in a DNS blacklist that really doesn’t belong there.

The reasons for this are varied. Direct reporting of a spamming IP address to a DNS blacklist may result in not just that IP but the whole netblock appearing on the list. Shared hosting suffers from a variant of this problem, as a single violating user can cause many sites to be blocked because they all originate from the same IP address. In other cases, end-users of large ISPs may decide to mark legitimate mailing-list mail as spam rather than unsubscribe from the list. Thus, that server may be blacklisted, at least from that ISP.

The lists themselves vary in focus and scope. The largest, sorbs.net, spamhaus.org, and spamcop.net, use general spamming guidelines to determine a host’s status. Rfc-ignorant.org goes a step further and lists mail servers that violate RFC 821 and 2821, which govern SMTP communication. Unfortunately, there are quite a few legitimate mail servers that violate these RFCs due to poor design and implementation, and anyone using those servers is likely to be listed by rfc-ignorant.org even if they’re not spammers. Certainly, those sites should be running compliant servers, but subscribing to this DNS blacklist can hamper otherwise legitimate communications.

That said, the most popular DNS blacklists have been honing their service over the past few years and offer significantly more accurate results than previous incarnations. In fact, services such as spamhaus.org and sorbs.net offer freely available lists that don’t just blacklist known spammer netblocks, but also list known dynamic IP netblocks used by carriers for home broadband connections, hosts running open proxies, buggy Web code that can be co-opted to send spam, and lists of hosts that have been identified as zombies and are spamming at the whim of a botnet controller.

How popular are these DNS blacklists? Steve Linford at spamhaus.org estimates that the spamhaus network receives between 80,000 and 100,000 queries per second, and that doesn’t count the number of large entities that don’t use the public servers, but have arrangements to pull the DNS blacklist databases to local servers on a scheduled basis, which significantly reduces the amount of queries to the public servers.

The cry of the wrongly accusedBut what about false positives? “Funny you should ask,” says John Shearer, Network Manager at Northfield Mount Hermon School. “Until last night we’d stayed away from DNS blacklists due to fear of false positives. In the past few months, however, we’ve seen a significant increase in our spam volume, and I finally implemented the njabl.org DNS blacklist in our mail filter. It’s stopped over 3,100 connections in the past 15 hours.”

Given the prevalence of DNS blacklists, false positives are always a threat, but the ever-growing spam problem is overriding those fears, as the benefits largely outweigh the negatives.

When a server is blacklisted, the site admins generally don’t know until rejected e-mails start bouncing back to users. In most cases, the bounced messages contain information on why the e-mail was blocked, and by whom. A URL is usually included in the warning message to instruct admins on how to request removal from the blacklist. Linford estimates that spamhaus.org’s turnover is 500,000 entries a day.

Each DNS blacklist uses its own method of collecting and maintaining its database. Many run honeynets that exist solely to catalog automated attacks from zombie networks, adding the source IP addresses to the database when they’re seen. Dead-end SMTP servers are also used. They don’t have actual mailboxes but simply absorb e-mail addressed to nonexistent users to identify spamming networks and systems.

Although the threat of open relays on the Internet isn’t nearly what it used to be, some still exist, and several DNS blacklists actively scan for open relays, blacklisting them when they’re discovered. It wasn’t long ago that many commercial SMTP servers shipped as open relays when used with their default settings. Today, that’s not an option. Nevertheless, John Gilmore, the fifth employee at Sun Microsystems -- a founder of the EFF, Cygnus solutions, and the father of UseNet’s alt.* hierarchy -- continues to run a restricted open relay. For him, it’s a free speech issue. For the rest of us, it’s simply bad practice and will render e-mail basically useless.

Floating in the grey areaGreylisting cleverly stymies the stupid bots responsible for most spam. The main functionality lies in an SMTP errorcode, which replies to the sending server to wait a few minutes before delivering the e-mail it just tried to deliver.

Normally, this errorcode is sent by receiving servers swamped with requests and can’t handle any more mail at the moment. Greylisting relies on the fact that most spamming servers and botnets try to deliver e-mail only once, ignoring the RFC requirement to retry delivery at a specific interval. For them to attempt retries on every e-mail sent would significantly reduce their overall volume.

Thus, every e-mail into a greylist filter is initially denied with the “Please try again later” errorcode. If the remote server resends the message in 10 minutes or so, the e-mail is passed through unmolested, as are any subsequent e-mails matching the headers of the first.

Greylisting has seen growing popularity recently. This spam-blocking method can significantly reduce the problem but also delays every e-mail until the sending server retries the message. The delay is necessary to separate the wheat from the chaff, but you will find that several legitimate senders -- and their ISPs -- have poorly configured mail servers, requiring that you bulk up your whitelist.

Nonetheless, greylisting, in concert with subscriptions to one or more DNS blacklists -- plus spam and virus filtering -- provide the key to relatively clean e-mail flows. It’s the rule for SMTP servers today. The chance of losing or missing e-mail is ever present but not generally a deal killer.

Toward a real spam solutionAs with any widespread technology, critical mass needs to be attained before any relevant change are seen.

One potential answer to the overall problem comes in the form of SPF (Sender Policy Framework). At its essence, SPF is a reverse verification performed on each inbound e-mail.

Just as every Internet mail server needs an inbound MX DNS record, SPF requires that each server maintain an outbound MX record, or rather, a notation in a domain’s DNS records that verifies that a certain server is responsible for sending e-mail. If a mail server using SPF finds that the sending server has no record in the domain DNS, then the mail is either bounced or at least marked as potential spam. For instance, if a message is received that claims to be from aol.com, but the sending server doesn’t exist in the SPF lookups for aol.com, then the e-mail is likely a forgery.

This solution has potential as well as caveats. For example, MTA (Mail transfer Agent)-level e-mail forwarding fails, requiring servers to re-mail, rather than forward e-mail to prevent problems with SPF filters. There are technical solutions to this, however, and they are in development.

Another option is secure SMTP using x.509 certificates. This method would require that every valid SMTP server on the Internet be assigned an identifying certificate. Only servers with valid certificates would be allowed to send mail to other servers. Again, this solution would require that the majority of e-mail servers operating now be assigned certificates and configured either to disallow uncertified delivery or at least put uncertified e-mail into probation.

Neither solution is likely to happen soon, although SPF has seen growing popularity recently. Until several major open source and commercial MTA products begin cooperating on a single standard, e-mail, as with the blacklists, will continue to be hit or miss.