Fingerprinting the World's Mail Servers

This summer, the sales staff at MailChannels came to the dev team with an urgent request: "Can you tell us which companies are running Sendmail? If we could know that, it would be so much easier to sell our Sendmail-compatible product."

For those of us who understand the SMTP protocol, the answer was, of course, a resounding "Yes." Most mail servers announce their identity when you connect to them on TCP port 25. The dev team decided that this was a summer science project they just had to get on top of. We even gave the science project a name: PingedIn, and we hope to provide more dynamic content on our skeletal website.

Stats Porn Teaser

Before I get into the nuts and bolts of querying millions of mail servers and fingerprinting them, I want to review some of the interesting results of the survey.

Our Survey Approach

First, a note on our survey approach.

One idea we had when we began this project was to survey all the mail servers in the world, using various domain databases as our source material. We actually tried this by downloading the dot-com and dot-net registries from Verisign and pinging the first few million domains.

The problem with this approach is that all domains are not created equal. It may surprise you to learn that speculators and fraudsters own most of the world's domains. When these people own a domain, it's rare that they provide email service on it (we found that only 10 percent of domains provided MX records). The more significant issue is that parked domains and phishing domains are not really a useful sample set for us to examine. Remember, our goal with this project was to find leads for our sales team.

Surveying all the mail servers in the world is also a daunting technical task. There are tens of millions of dot-com domains--never mind dot-net, dot-org, and all the other TLDs. The data from a survey that large would fill terabytes of disk space, eat up monster bandwidth, and starve an already lean and mean startup of much-needed capital resources--not to mention taking many weeks to complete.

Rather than surveying all the domains in the world--which would have been a fun project--we chose instead to survey only those domains that have a real company behind them. We partnered with an old-school company data firm to get a list of 400,000 companies worldwide as our source material. It's not the perfect solution, but it suits our needs, and we can survey all 400,000 in a couple of hours without so much as raising a single abuse complaint.

With this little detail aside, where's the stats porn?

Open Source Still Dominates

Open source still dominates the global mail server software market. The changing nature of email threats such as spam and viruses are causing many companies to install an extra layer of protection at the network edge: witness Postini's rise to nearly 10 percent market share (Figure 1).

Figure 1. Open source mail server software dominates the market

Of the 400,000 domains we surveyed, 31.2 percent of them (still) receive their email via open source mail server software. Of these, the most popular by far is still the old guard, Sendmail (12.3 percent), with Postfix a relatively close second (8.6 percent). Exim and qmail are roughly tied (5.3 and 5.0 percent, respectively) in third place.

It's hard to tell what email security software lies behind the open source offerings, because generally these are behind the scenes where our pinger can't detect them. But it's a fair bet that many of the open source installations run the venerable SpamAssassin and that the remainder work with open source-friendly vendors such as Sophos, Proofpoint, and Symantec (Brightmail).

Next Up: Hosted Services

A surprising result of our survey has been the emergence of hosted email security services. These services prefilter email traffic for a domain before passing it on to the receiver's destination mail server--supposedly free of spam and viruses. Customers of hosted services pay a premium to get rid of their email security hardware, but the performance of these services is reportedly quite good.

Postini is the clear leader here, having secured 8.5 percent of the domains we surveyed. Next up is MXLogic, with 6.0 percent. Then there is Concentric Hosting with 4.5 percent, Earthlink with 2.7 percent, and Yahoo with 1.0 percent. The rest of the service providers own chunks too tiny to mention here.

The Evil Empire

Microsoft is in fourth place, with Exchange taking 7.6 percent of the domains we surveyed. You might think Microsoft would be doing better, considering that their web server software represents 31 percent (according to Netcraft's November 2006 survey). This weak result shows that Microsoft has a long way to go before it can establish itself credibly as an email boundary vendor.

Toasters, Microwave Ovens, and Other Appliances

There has been a lot of buzz over the past two years surrounding the rise of network appliances. The email space has not been immune to this trend, and now almost everyone seems to be getting into the appliance game. However, despite the buzz, the penetration of appliances remains very small indeed. The most significant appliance vendor in our survey is Barracuda, with a mere 2.8 percent of the market. IronPort is in second place with 0.8 percent. And Ciphertrust (now a division of Secure Computing) has 0.6 percent.

The appliance vendors will probably comment at the end of this article that while they don't own much of the broader market, they do own a significant chunk of specific segments--such as the Fortune 500. I'll get to that later.