QuoteReplyTopic: Greylisting story Posted: 22 December 2007 at 10:05am

Hello all,

Summary: Please give us greylisting capabilities.

Details:

We have been in deep s**t since the past 5 days. Our servers were
hammered with spam 4x 5x times more than usual and our two SFI servers
were not able to handle the load at all.

Note that before the SFI servers we have a firewall that blocks discriminatively half of the world countries and then it blocks any hosts/networks that were blacklisted by SFI and still tried to connect 10 times to it. (we get this from the logs)

Adding more servers will basically open more doors and the situation
would stay the same, or even worse the backend (quarantine database)
will also suffer more.

Our sysadmin has talked always about greylisting and I have been waiting to try it when SF implements it because we are using all the cool features of SF and I dont want to re-invent them.

There are many softwares for doing greylisting and using them means
that I will have to re-implement spf checking, rbl checking, keyword
checking, lose sfdb, etc.

Another possibility is to use greylisting proxy that forward the good
requests to spamfilter but then we lose the origionating IP address and
most of the checks will not work (rbl, sfdb, spf, etc)

We have tried greylisting on our corporate domain which we separated form SF for a day to first get the taste of unfiltered mails (spams). We got 25K (TWENTY FIVE THOUSAND) in our catchall mailbox in a day. After we just put a greylisting proxy in front of it (our corp mailserver was not doing, rbl, spf checks anyway. It was implemented with "SF will do the filtering and will only deliver good mails to this server" in mind). Anyway after the greylist proxy implemented we only recieved around 800 (EIGHT HUNDRED) messages in the catchall mailbox. Not bad!

During all these experiments, our ISP customers were still suffering because mails were not delivered (recieved by SF because it was too busy), etc, etc.

Solution: We have implemented a couple of firewall/greylisting servers.These 2 servers replace the firewall that was sitting behind the SFI servers.Each SFI server uses one of them as their default gateway.

The system runs postfix, knows which domains and mailboxes we accept mails for.We use sqlgrey (a greylisting plugin for postfix written in perl that reads/write connections and whitelists from a mysql or some other database ) for greylisting.

The database server is a central server that is also used for mailrouting, maillogging etc, so more than one instance of this application can use it.

Basically this server will tell everyone (except a few IP addresses of our choice) to come back in 5 minutes on their first connection. 95% will not come back (zombies, hacked machines, other smapnets, etc). The other 5% when will try again after 5 minutes, the mail will be accepted and forwarded to the mailbox (Yes the SPAM will get through). At the same time the application will add this ip address with the user/domain (tripplet) to a from_awl table.

Nothing special so far. SF is not getting any mails at all. Every 5 minutes we have a script that looks in the from_awl table for entries in the last 10 minutes and add a NATing rule to forward that IP address or the subnet to the SFI server.

This means that every IP that the greylisting accepted now goes no more to the greylisting server but to the SF server.

Results: Before this trick we had between 600 to 1000 connections on the SFI servers at almost all the time.After this trick with 35K ip/networks in the list (known rfc compliant mailserver/persistent spammers) now go directly to the SFI servers and I have not seen the connections go higher than 10 (TEN). YES really I was sure that something is screwed up somewhere but its not. We have mails flowing in again normally.

Ok Atif... if you ask this way... you're making us spoil the surprise...We're alpha-testing the new SpamFilter ISP v4.The main two features are two new filters.

1 - SpamFilter Distributed Content. SpamFilter will have the ability to detect similar emails, and will create hashes (signatures) for each group of similar emails. The signature will be uploaded to our centralized database, much like our SFDB. If we receive reports for the same hash, but the emails that have this specific hash are originating from different IPs, we will consider this hash to be a spam hash. From then on, any emails with the same hash will be thus rejected (the theory is that legitimate servers will send their newsletters and mailing lists from the same origin IP, not from different networks... Thus if the same email (or similar - as again SpamFilter is able to group together emails that have similar text in them) is being sent from multiple networks, chances are it is not legitimate.

2. GREYLISTING (but as usual it will be a LogSat's "flavor" of greylisting...)

If anyone is interested in the beta, please email us with your order number and, most likely starting from tomorrow, we'll be able to provide it to you (with an option to enable the greylisting as it's turned off by default).

I read from one your posts "Our only option would be to implement greylisting, but only for single-server configurations"Is this still the case, or did you find a way to sync it between all servers. (sqlgrey simply puts everything in the db and make lookups there).

I will send you an email with the order number for the alpha to test the greylisting.

For the SFDC: Is is similar to DCC (http://www.rhyolite.com/anti-spam/dcc/)If yes, why re-invent it instead of implementing a dcc client in SF?

In this beta, the "GreyListAllowed.txt" file that is used to hold the list of IP addresses that has passed the greylisting stage and are allowed to make connections is self-maintained and handled by the running SpamFilter. It cannot be changed by external application.

In the next betas we'll work with SpamFilter Enterprise to have this list stored in the database and distributed amongst the various SpamFilters.

Installed beta over a SFE.730 Installation on our testing server. Running MSSQL 2005. SQL is located over a WAN link.

Upon first start of service, machine cpu goes to 100%. I let it stay there for about 20 minutes. Ram usage on that process was changing. After 20 minutes, I gave up and killed the process. Started the service again and same result.

GUI does not show, however tray icon does show.

SFE process using 35,000 K of Mem with 6 threads

Database stats:

bl_ips - ~200K rows

domains ~100 rows

authorizedUsers - ~3000 rows

bl_domains - ~5000 rows

All domains using same settings

All other tables are small (except for tblmsgs and tbl_quarantine)

Normal startup on SFE.730 would take about 30-60 seconds on this same machine.

now as writing this, GUI did show, but still not accepting connections and CPU still at 100%.

At about 40 minutes, gui did start to show connections and CPU went back to a "normal" usage.

It seems to me that the slowness results when SFE is reloading the larger tables from the database. I did go ahead and install the beta on our primary server (same server that is running the sql) and it behaved the exact same way as the testing server. Took more than 10 minutes to begin accepting connections and show the gui.

I also noticed that each time the tables were automatically reloaded, the CPU on the testing server will go to %100 for 2-5 minutes and then resume back to normal. During this time, it did gontinue to process connections though.

It may take up to a minute to load tables on the 3.5, but much much longer on the beta. Once it has loaded, It appears to be fine (until it reloads a table).

Roberto - 2 beta issues:

I am getting a fair number of false positives on the SFDC filter. Out of ~500 in the quarantine, I quickly found 14-15 emails that should have been delivered. Is there anything that I can do that will help you with this? (Actual Emails, logs, etc)

Also, my primary server did quit accepting connections today. It did continue to send quarantine-force-delivered emails and perform tasks (such as cleanups, corpus, etc). I restarted the service and as of 15 mintues - still no GUI. Want logs?

We have moved our corporate domain greylist server to SFI beta and its performing fine.

We have also moved the 2 SFI server for our isp platform to the beta thus removing the greylist before NAT solution that we implemented temporarily.

The greylisting is now being handled by SFI.

The two firewalls (one each for SFI) are also removed and replaced with the old firewall which again denies access (based on country and on number of attemtps on SFI after being added to local blacklist cache ) before passing the packet to the SFI.

just installed the beta and am very impressed! the performance gain is great, before I usually had 10 - 15 concurrent connections which had to run through the various filters and now I'm down to 1 - 2...

I just installed the beta yesterday and our quarantine levels are down 80% per hour. If you are using MYSQL -- here is a way to get a count per hour on qyarantined messages:

select MsgInfo, Count(MsgHour) MsgCountfrom (select DATE_FORMAT(MsgDate,'%m-%d-%Y %h %p') as MsgInfo, DATE_FORMAT(MsgDate,'%Y%m%d%H') as MsgHour From tblquarantine) as msgtableGroup By msgtable.MsgHour Order By MsgHour Desc

Very impressed thus far. I also sent an email to my hosted clients announcing the greylisting as a positive step in spam reduction and to expect an email to be delayed a few minutes. I've had several atta-boy's from the clients responding to this email.

>>In this beta, the "GreyListAllowed.txt" file that is used to hold the list of IP addresses that has passed the greylisting stage and are allowed to make connections is self-maintained and handled by the running SpamFilter. It cannot be changed by external application.

In the next betas we'll work with SpamFilter Enterprise to have this list stored in the database and distributed amongst the various SpamFilters.

I hope that this does not mean that we need to run SF in enterprise mode with the database enabled as we run SF in SFI mode, forwarding emails to an internal server??

Just wondering if I am the only one seeing a large (huge really) decrease in ATTEMPTED connections ... almost like the Spammers add my servers to a "don't bother trying again" list. I do know that many applications will add servers to a "suppression" list but usually only after getting a "hard" bounce - 5xx code.

Anyway, after a large amount of log parsing I have set the following settings and am really enjoying the results.

I hope that this does not mean that we need to run SF in enterprise mode with the database enabled as we run SF in SFI mode, forwarding emails to an internal server??

Do not worry, with SpamFilter ISP "standard" we will not be requiring the use of the database for anything else other than the quarantine database. All filters will always work without the need of the database.

First, I want to state once again, how effective the GreyListing has been for us. Here are some observations after MANY hours of log analysis.

Over the first 6 days, our inbound connection attempts went way up as most messages had to do 2 attempts as expected. Over the 6 days, the connection count went down and became asymptotic as the GreyListAllow list populated ... again, as expected. As of today, our quarantined items has reduced to one quarter, dramatically reducing the load on our SQL server while the actual, delivered good mail quantity remained at it's normal levels. After the first few days I relaxed several of my RegEx filters based on False Positive reports (automatically generated every time a customer pushes a message out of quarantine).

Bottom line is that we have reduced our False Positives to about 0.0095% while our server resources has been reduced to about 1/5 of the level it was running prior. Oh, and customer complaints ... ZERO!

Yep, same positive results here after about 6 days. My own email quarantine was averaging about 60-70 new quarantined items a day and now its about 3-4 a day.

So the greylisting picks up the single shot spammers and the blacklist cache cleans up the repeat offenders who got thru the greylist. A good 1-2 punch that spammers will find hard to get around. And with the SFDC cranking up V4 SFE is getting to be an even better product. Thanks for listening and improving it!!

You cannot post new topics in this forumYou cannot reply to topics in this forumYou cannot delete your posts in this forumYou cannot edit your posts in this forumYou cannot create polls in this forumYou cannot vote in polls in this forum