For those of you interested in the scripts I have posted so far that have figured out how to use them, a false positive presents a problem, removing the false positive address from the file. My address list has close to 2500 addresses, 2400 of them added manually before I wrote the script. As such, there seem to be no false positives [yet]. Now that the process is automated I anticipate it might happen occasionally [to others <g>].

The script below will ease removal of the IP address of the current false positive message from SpamAddress.txt (or some other file). This script is intended to be run manually, such as from a button, not automatically, at least not without some modification.

The line numbers where the address is found can give some indication of when the most recent and oldest use of this address occurred. this isn't really necessary here, but I wanted to debug the technique for use elsewhere.

{ RemoveIPAdrFromFile - Version 1.10{ Author: Scott Taylor - January 24, 2006{ Also see scripts DisplayMostRecentIPAddress, MoveIPAddressToStartOfFile{ and AddIPAdrToFile (formerly AddIPAdrToAdrFile).{{ Purpose: Remove all occurrences of the most recent IP address in the current message from the file containing{ banned IP addresses. Note: This program is not limited to only using this file, but it is the original{ purpose of this script. This script is intended top be run manually such as from a button.{{ Method: The script gets the most recent IP address of the current message, opens the specified file and{ removes all occurrences of that IP address from the file, overwriting the original file with the remaining text.

{ initialize variablesSet #n 0 { counter to track how mnay times address is foundSet #x 0 { working copy of #nSet $a "Newest occurrence found at line "Set $b "Oldest occurrence found at line "Set $c "Single occurrence found at line "

:LoopDoneAddIntegers #x #n { add how many lines have been deleted (added v1.10)Dec #x { do not include the current line (added v1.10)AddStrings $b #x { save last occurrence in string $b (oldest){ Comment out the next line if you want to locate the addresses and not remove them (added v1.10)SaveBody $ExistingAdrs $FileName { Save remaining text, overwritting the old file.GoTo finish

:finishSet $TmpName $Filename { make a working copy of the file nameChopString $TmpName 1 3 { get rid of the leading "..\" to reduce confusion of messageSet $msg "Found "AddStrings $msg #n " occurrences of IP address " $MostRecentIPAddr " in file " $TmpNameIf #n ! 1 Then NotSingle { see if #n is not equal to 1 InsertLine $msg 99 $c { add single line number to end of message GoToSkipLine

:NotSingleIf #n < 2 Then SkipLine InsertLine $msg 99 $a { add newest line number to end of message InsertLine $msg 99 $b { add oldest line number to end of message

:SkipLineMessageBox $msg { display informative message:Exit

Last edited by FieldDir121 on Wed Jan 25, 2006 8:58 am, edited 3 times in total.

After I posted the code I realized that by commenting out a single line the same code could be used to locate the addresses without deleting them from the file on the disk. This might be useful to determine how long ago the address was used.

For instance, if you get 50 spams per day and the address is found in the first few hundred addresses the message isn't very old. If the address is found at line 1857 then it is quite a bit older.

What you are doing should work. I used to use 'Entire Message' but decided 'Message Headers' will work just as well and may be a little faster for messages with large bodies.

Initially I deleting message caught by the filter. Occasionally I sent them to the junk mailbox. After a while I wondered how many messages were being caught by this filter so I created a special mailbox, SpamAdr. Now I can see for sure. The price is having to manually delete them.

At first the IP address filter was catching the majority of the incoming spam. I have another filter using the same technique that uses keywords and phrases, SpamEntire.txt. As that list has improved the number of messages that make it to the SpamAdr mailbox has decreased. Keyword messages are placed in the mailbox SpamEntire.

I was wondering if anyone was utilizing what I have done so far, wondering if I should continue to post the scripts I am working on. One person is enough since it doesn't take much extra effort to share.

My current effort is a script that will search the SpamAddress.txt file for the IP address of the current [incoming] message and insert the line number(s) where the address is found at the beginning of the message, InsertAdrLineNumberIntoBody.

The technique is only difficult because html and plain text must be handled differently. Determining which type is the tricky part. I just found a script by Pete that adds attachment names to the message body. He had one additional test criteria than I did. I added it this morning. After some additional testing I may be ready to post the result.

This new script will allow me to quickly determine how many of the messages in SpamEntire would have been caught by SpamAdr if they hadn't been caught by SpamEntire first. If the IP address is not found in the SpamAddress.txt nothing is added to the message body. This makes the yes/no determination very quick.

Another interim script will be to add the current IP address to the beginning of the SpamAddress.txt file and delete any other other occurrences of the address from the file. Otherwise, some addresses could (and do) appear many times. This will also put the most recent and most active addresses at the beginning of the file.

Since some addresses used by spammers will be spoofed, not their actual address but the address of an innocent, periodic manual truncation of the SpamAddress.txt file from the end will eventually remove addresses used only once.

Eventually I plan to integrate several of the scripts plus a few more I have in mind into one or two more comprehensive scripts, assuming I don't run out of spare time first.

For instance, once duplicate addresses are eliminated from the SpamAddress.txt file, there will be no need to display more than a single line number of where the address is found.

I wasn't able to get Excel to sort IP addresses. I didn't try very hard though. I originally wanted to sort numerically but eventually decided that sorting by most recent might be more useful. That is why I gave up on using Excel to eliminate duplicates.

>> InsertAdrLineNumberIntoBody?

What do I do with it?

Why put the [spam] messages in their own mailbox? Same idea. I want to see positive results. If catching a few more percent of the spam messages using an automated technique, spam addresses, is possible I would like to do it. I can add that technique to the system used by my wife and kids. If I have to maintain it manually it will not be very up to date. I delete the messages once I look to see if they have an IP address inserted or not. The line number where found indicates how long ago the address was last used. This gives me some idea of how long addresses not recently used should be kept.

I was even thinking along the lines of writing a script that identifies a specific attachment. When that attachment is found in an e-mail to one of the accounts on that system the script would replace the filters.ini, SpamAddress.txt, SpamEntire.txt files and any scripts. This is the height of lazyness since I walk by the room with that system many times each day.

{ InsertAdrLineNumberIntoBody - Version 1.01{ { Author: Scott Taylor - January 29, 2006{ Also see scripts DisplayMostRecentIPAddress, MoveIPAddressToStartOfFile, RemoveIPAdrFromFile{ and AddIPAdrToFile (formerly AddIPAdrToAdrFile).{{ Version 1.01: Reformatted comments so they wouldn't wrap when posted in the Pocomail forum.{{ Notice: I have done some testing on incoming mesages. This script appears to work properly for{ the types of e-mails I receive. That doesn't mean it will work properly for all e-mails or any{ of your e-mails. Problems may include, but are not limited to, complete loss of an incoming{ e-mail and/or corruption of the e-mail contents.{{ Purpose: This script is for informational purposes. Inserting a marker, in this case the line{ number in Spam Address.txt, at which the IP address was found, provides a visual indication of{ messages that would have been caught by SpamAddress.txt and the associated filter if the message{ had not been caught by another filter first. The actual line number provides some indication of{ how recently the address had previously been used. Since these messages will be ultimately be{ deleted, altering the contents is acceptable (in my case). Ultimately the goal is to increase{ the chances of catching e-mails from repeat addresses that are not caught by other filters.{{ Method: This script gets the most recent IP address of the incoming e-mail message and a copy of{ the body of the message to work with. SpamAddress.txt is searched to see if the IP address of{ this message is present and at which line number(s). Each line number at which the address is{ found is tghen inserted into the beginning of the incoming message body. The working copy of{ the message body is then used to replace the original message body and processing continues as{ if nothing has happened.{{ HTML versus non-HTML are both accomodated. If the incoming IP address is not found, or any{ similar IP address errors occur, the script exits leaving the message untouched.{{ This script doesn't change the ultimate destination of the incoming message. Also, it won't work{ on messages that are already in a mail box. It will work on messages that are being moved from{ one mailbox to another, such as by a filter.

{ Get the message body from the current incoming message (raw allows html or text only format).ReadRawBody $body %message { get message bodySet $LowerBody $body { make an expendable copyLowerCase $LowerBody { set to all lower case to make searching for strings easier

{ Determine if HTML or plain text. My original method evolved to something close to this. A few{ messages were still getting through incorrectly identified. Attachments Lister.poc by Pete had{ a similar approach but appeared more comprehensive so I copied that portion his code here.Set &noHTML false { default to HTML styleSet $LineTerm "<br>" { set line terminator to HTML styleReadHeader $contentType "Content-Type:" %messageLowercase $contentTypeIf $contentType = "text/html" Then UseHTML

Here is the script that will delete all occurrences of the IP address in a file and insert a single copy as the first line of the file.

It can be tested by using AddIPAdrToFile to put multiple copies into the file. LocateIPAdrInFile will then show how many copies exist and the line numbers of the first and last occurrence. After running this script LocateIPAdrInFile will indicate a single copy at line 1 (remember line 0 has been intentionally left blank).

{ MoveIPAdrToStartOfFile - Version 1.00{ Author: Scott Taylor - January 30, 2006{{ Also see scripts DisplayMostRecentIPAddress, InsertAdrLineNumberIntoBody, RemoveIPAdrFromFile and{ AddIPAdrToFile (formerly AddIPAdrToAdrFile).{{ Purpose: This script has the same result as if RemoveIPAdrFromFile had been run followed by{ AddIPAdrToFile, without the message box that requires human intervention. The result will be that{ all copies of the IP address of this message will be removed from the file, SpamAddress.txt, with { a single copy of the address being added as the first line of the file. Eventually, addresses not{ used for the longest time will be at the end of the file.{{ Method: The script gets the most recent IP address of the current message, opens the file{ SpamAddress.txt and removes all occurrences of that IP address from the file. A new copy of the{ address is inserted into the file at the first line.

Wanting to know how well my filters are doing, the next script I work on is likely to be one that allows statistics to be derived. I was thinking of updating various information in a text file each time a message comes in. The actual percentage math, a ratio of spam to non-spam, may not be included in the script, leaving that for a handy calculator. The reason for this is explained below.

The first script will count the total number of incoming messages, either on all accounts or on a specific account. A second script will count how many messages are stored in the SpamAdr mailbox (the mailbox names can be changed to the names you use). Either the script will have to be run after the message has been placed in the mailbox, assuming it isn't too difficult to determine the mailbox name within the script, or a slightly modified version of the script will need to be created for each mailbox, ie., SpamAdr, SpamEntire, Junk Mail and the main mailbox.

Here is what I am thinking of having so far:
nnn Total number of incoming messages
nnn Number of messages placed in SpamAdr
nnn Number of messages placed in SpamEntire
nnn Number of messages placed in Junk Mail
nnn Number of messages placed in UnKnown Sender

The way I have my system configured, only e-mails from people in my address book are guaranteed to be placed in the main mailbox. Those not placed in the main mailbox and not determined to be spam are placed in the Unknown Sender mailbox. I have to manually look through these for non-spam e-mails.

This arrangement makes determining the spam to valid e-mail ratio a bit more tricky, especially if messages are deleted directly out of Unknown Sender. Those messages will not be added to the other counts. This is one reason I may not add an automatic ratio calculation.

What isn't clear, yet, is which column, if any, to add the number of spam messages from the Unknown Sender mailbox. They didn't get caught by any filters and they weren't put in the main mailbox. Adding them to any of the above catagories after manually sorting the messages in Unknown Sender will distort the results. Perhaps using a third script, activated by a button, that increments another catagory:
nnn Manually designated spam from Unknown Sender

On my system a script counting the messages actually placed in my main mailbox would need to be called from several filters, so I may not include that, instead figuring anything not put somewhere else ended up there.

Comments and suggestions welcome as I haven't started this script yet, and it will more than likely evolve as time goes on.

I have received around 100 spam e-mails since I put this script into use. Today was the first time an HTML e-mail appeard to have been significantly altered, other than adding the intended text. I think the problem has to do with the text being added above the HTML header.

If someone knows why things are not as expected I would appreciate comments. Otherwise, since the e-mail will be deleted anyway I may not spend much time revising the script.

Using the AddIPAddr script, I have built a listing of over 250 unique IP addresses. However, as suggested earlier any match against these IPs is routed to a specific folder. Although I have since added duplicate IPs, not a single message is being filtered based on IPs. In the filter list, I have this one set as the FIRST filter...

That got me to thinking, Scott, you add [] to the IP address yet that would NEVER be found in the headers. To test this, I deleted all []'s in the text file but for 48 hours, no hts, yet more duplicate addresses have been added.

If you turn on the header you will see that the IP address is [always] enclosed in square brackets.

As I mentioned earlier in this topic, or perhaps another, you need string terminators at both the start and finish of the address. Otherwise you could get unexpected results.

12.1.1.192 would also appear to be 112.1.1.192 and 212.1.1.192. A similar case exists for the end of the address string, 192.168.1.1 will be a hit for 192.168.1, 192.168.1.10 through 192.168.1.19 and 192.168.1.100 through 192.168.1.199.

I have a separate filter watching for groups of addresses:
[60.
[61.
etc.
I keep these in a separate file and use a separate filter. Individual IP addresses are considered definite spam. I am sure enough that at times those messages are automatically deleted by the filter rather than stored. The same goes for messages in SpamEntire. I look through the SpamBlock messages to be sure there are no legitimate messages.

As to not getting many hits by SpamAddress, my SpamEntire filter catches about 2/3 of the incoming spam. 1/3 ends up in the Unknown Sender mailbox. SpamAddress gets an e-mail every day or two. I am hoping my recent automation of adding IP addresses to SpamAddress.txt will improve this. I stopped manually adding addresses many weeks (months?) ago.

SpamAddress.txt has grown from 38kB to 48kB in the last couple of weeks. At a maximum of 19 charaters per line, "[nnn.nnn.nnn.nnn]<CR><LF>" that is over 500 new IP addresses. For the past couple of days duplicate addresses of hits are also being removed. Note: Existing duplicates remain in the file until a new message arrives using that IP address.

I attribute the lack of benefit form SpamAddress to the success of SpamEntire rather than to the failure of SpamAddress. My justification for this view is that early on SpamAddress caught more than SpamEntire, which comes first. I do this because keywords can catch spam from many addresses. IP addresses can only catch spam from a single address.

For any message that makes it to SpamAddress I manually extract keywords. That means that next time the message that made it to SpamAddress will have been caught by SpamEntire, assuming there were unique keywords.

About half of the messages that still make it to Unknown Sender have no unique keywords. To me unique means they are unlikely to ever occur in a legitimate message, hence my to date 0% false positives.

Please copy the entire path and file name from the "for" and post it. Also, a screen grab of the root directory of your F: drive. No insult intended, but let's eliminate the obvious first.

Here is one of mine:
%file%:"SpamAddress.txt"

Notice that I have "file" where you have "filename". Not sure if that matters or not. See the 7th message in this thread for a link to the FAQ about using a file within a filter.

Next a screen grab of your filter screen.
(Note from just after I posted this: A screen grab might show too much of your personal information for an open forum like this. What I was after was other filters that might affect where the message ends up.)

Question: Do you now have one mailbox called "Junk Mail" and another called "Junk Mail\Spam IP Addre..."? (... since I cannot see what comes next)

If so, just to eliminate a possibility, change the name of the "Junk Mail\Spam IP Addre..." mail box to something that does not contain the exact phrase "Junk Mail".