Wanted: Postini anti-spam trialists

‎13-11-200712:43 AM

*For the latest Postini anti-spam developments please see the more recently published update that can be found here*
Spam is undeniably one of the biggest challenges we face as a service provider when it comes to building a reliable, stable and dependable email platform that our customers can rely on. Some have even argued that the ever increasing torrent of unsolicited email that now plagues the Internet has almost brought question to the usefulness of email as a reliable tool altogether.
Providing bandwidth, dealing with all the problems caused by spam (not least mail delays!) and maintaining constant house-keeping regimes is extremely costly. Spammers are continuously changing their techniques to circumvent the anti-spam precautions providers put in place, and things are made extremely difficult due to the lack of consistency in the way different email servers are set up around the globe.
So what does this mean I hear you cry? Are you turning email off? Well, fortunately enough I don't think it's quite come to that, although we would like to ask for your help...
As some of you will be aware, the last few weeks have seen us trialling Critical Path - a third party anti-spam appliance, two of which now sit in front of our primary mail delivery servers helping to manage load across the platform. Those who've been monitoring the progress of this work will know that it's gone relatively well, but it's only the start of our plans for the mail platform and the battle against spam is certainly far from being won.Towards the end of the last announcement I mentioned that we had been looking at a number of other anti-spam/email secuity solutions with a view to implementing further trials. One of these solutions is provided by a company called Postini.
We've actually been using Postini's Email Security solution for a while now on our internal mail platform, and we've since reached the position where we'd like to look at offering this to our customers. The way we'll be implementing it is to first trial the solution with a small subset of customers who can help us test the platform. If you're interested in taking part in this and think you could offer some worth then please read on to find out how you can register your interest...
Some of you will already be familiar with Postini and may have encountered or used some of their offerings in the past. They're a global leader in on-demand email, Internet and security solutions and were recently acquired by Google for a cool $625 million. More than 35,000 businesses worldwide depend on Postini. They process over 1 billion messages a day and have been rated as the top message security vendor in independent magazine, test lab, and analyst reviews. Impressively, Postini process more messages each day than all other managed security service providers combined and boast 99.999% reliability.
During our internal trials, we've found Postini to be *very* good at blocking spam at the 'edge' of the network. This is something that could help us massively in the run up to Christmas by significantly reducing bandwidth, CPU cycles and load on our mail servers (and the Critical Path boxes). From a customer perspective, it means you'll see a reduction in the actual volume of junk messages you receive and it should also help improve the reliability of the spam filter.
What will Postini do?
Unlike Critical Path which is a physical network appliance, Postini's Email Security is an 'on-net' solution based on DNS. The benefit of this is that no additional hardware or software is required. There's no ongoing housekeeping or maintenance to worry about and all updates are done automatically. Whilst we'll not be offering it initially, introducing Postini will also eventually allow us to offer a much wider and intuitive array of spam-filtering controls. Things that customers have been asking after for a long time, like per user whitelisting, blacklisting and quarantining.
Initially we will not be tagging emails as [-SPAM-] however we do intend on introducing this before we push the solution out to non-trialists. We will probably look at enabling subject line tagging for both Postini and Critical Path at the same time as it's essentially the same change that needs to be made to our mail platform. This means that to begin with the only noticeable difference trialists should see is a general reduction in the amount of spam they receive.
When subject line tagging is switched on it will be applied in accordance with customers' Manage My Mail anti-spam preferences and deleted, delivered to your inbox, or delivered to your 'Spam' folder as required.
Messages will still go via Critical path, Clam & Dspam for the time being however this is subject to change depending on the success of the trial.
How can I take part?
You can register your interest by visiting http://trials.plus.net and clicking on 'Please add me to the Postini anti-spam trial'. You should then follow the on-screen prompts in order to raise us a ticket. The intention is to migrate customers in batches so assuming you are accepted onto the trial you should receive a response within 5-7 days.
The proposed timeline of events is as follows (some of which has already been done):

Roll-out to internal mail platforms and implement subject line tagging

Roll-out to staff broadband accounts minus subject line tagging

Roll-out to trialists minus subject line tagging

Introduce subject line tagging to trialists

Roll-out to non-trialists complete with subject line tagging

Roll-out per-user management to trialists

Roll-out per-user management to non-trialists

When we get round to rolling the changes to the live platform, users will be batched up into groups of 50,000 or so. Each day a batch will be migrated, with the entire customer base expected to take around 5 days. A Service Status covering this work will be posted nearer the time.
If you do choose to take part then please take the time to read this post in full as we will be unable to answer any specific questions about the trial via the ticketing system. If you do have a question you would like to ask before taking part then please visit our discussions forums where me and the rest of the Comms team will be happy to address your enquiry. The forums are also the best place for you to leave any feedback about your experiences on the trial.
The roll-out
The technical aspects of this work will involve Exim configuration changes, DNS/Database alterations and some script development.
The DNS records of customers' domains will be changed so that inbound email traffic is directed to an anti-spam proxy platform run by Postini. Postini will then hand this email off to Critical Path and our mail delivery servers (the mx.cores and mx.lasts). The second phase of the work will then be to block any messages intended for customers that have not come via Postini. This will be email that spammers are sending directly to the mx.core or mx.last virtual hosts.
The work can be split up into the following key deliverables:

Change the DNS update processes

Change the email server process to block non Postini routed messages

Create automated processes to populate the Postini system with customer domains and mailboxes

Implement a manual process to handle trial users

Manage customer expectations carefully using detailed Comms, Community support and the slow migration of users

The following two diagrams show a before an after view of the mail platform so you can see where Postini fits in. Bear in mind that the Critical Path boxes don't appear on these pictures despite the fact that they're currently housed behind the load balancers.
Before:After:Risks
As is to be expected there are a number of risks associated with implementing a piece of work like this. Whilst we will obviously do our utmost to safeguard against these, it helps that we make customers aware of the implications of the work. Especially those wanting to take part in the trial:
Size of plus.com zone file - This is the actual file which holds the DNS information for all people with addresses under the plus.com domain and it's now quite large. There have been instances in the past where Bind (our DNS server software), has had problems loading this file. As part of the Postini work, we'll be adding four entries for each user, and only taking away two so we could end up creating a situation where Bind fails. To help mitigate this risk, one of things we'll be doing is to update one authoritative DNS server at a time during the migrations. This allows us to abort the update, and switch to an alternative approach with minimum impact on customers should the worst occur.
Back to back trials - The fact that we're trialling multiple solutions at the same time has the potential to cause delays when diagnosing issues. This is why we've allowed plenty of time for Critical Path to bed in and why we're offering Postini on a trial basis to begin with.
Spurious domain entries - If a domain that isn't hosted on our DNS platform is somehow still present in our mail platform then it will stop working once we start blocking messages that haven't come from Postini. Basically this shouldn't happen, however if it does we'll be able to quickly reverse the blocking on a per-user basis.
So that just about sums it up! I'm aware it's been another long read so if you've made it this far then congratulations. All that's left to do now is head over to http://trials.plus.net to register your interest
Bob Pullen