Detecting forged mail sent from "yourself"

Spammers know that you probably don't want to reject mail that you sent, so one technique they sometimes use is to forge the sender address so that it looks like you are sending mail to yourself. To reduce the chance of the whitelisting rules applying to spam messages like that, we try and detect the case where the sender address and recipient address are the same, and skip the whitelisting test in that case.

Whitelist checking algorithm

While detecting if the sender and recipient addresses are the same sounds easy, it actually gets complex in the general case when you want to try and handle email sent through a forwarding service, or retrieved via POP.

A full description of the algorithm used for matching emails against your address book for whitelisting is included below to help with debugging edge cases.

Build a list of sender addresses by looking at the email address in the SMTP MAIL FROM envelope (which we attach as the X-Mail-from header on every email), the From header of the email, and the Sender header of the email.

Remove from the sender addresses list any address if it's in a Resent-from header.

Build a list of delivery addresses by looking at the Received headers for text of the form: for <user@example.com>.

Remove from the delivery addresses list any address if it's in a Resent-to header.

Some systems (outblaze, NY Times) annoyingly use the for <user@example.com> incorrectly, and put the sender rather than the recipient, so we treat these known bad senders specially and ignore the appropriate headers. (For implementors, RFC2821 says: "The FOR field MAY contain a list of <path> entries when multiple RCPT commands have been given", making it clear this should be the recipient address, not sender address.)

If the final address in the delivery addresses list is not the final recipient address at FastMail (e.g. the SMTP RCPT TO address used when the email was sent to us, which we put in the X-Delivered-to header), then we add the final recipient address to the list of delivery addresses.

If there are any delivery addresses (usually the case), then we iterate over each address in the sender addresses, and skip it if it exists in the delivery addresses list, otherwise we check it against your contacts list.

If there are no delivery addresses (email was pulled via POP, and there's no for <user@example.com> items in Received headers), we iterate over each address in the sender addresses, and skip it if the address corresponds to one of your personalities, otherwise we check it against your contacts list.

One suggestion has been just to ignore the address book whitelisting for any From address that's also one of the user's personalities, rather than using that only as a last resort. The problem with this is that in certain common cases it causes things to be not whitelisted that should.

For example, some people have a separate second mailbox, and set that mailbox up to forward to FastMail by default, but sometimes use the second mailbox. For example, you might have forwarding from joeblogs@example.com to joeblogs@fastmail.com, and also have a joeblogs@example.com personality. If you happen to be at example.com, and send a message to joeblogs@fastmail.com, the personality approach wouldn't whitelist the email from yourself. This happened quite a bit, and caused problems for quite a few users.

Another separate example, many businesses have a shared "sales" or "support" mailbox. Each user has their own personality, but also creates a "sales"/"support" personality so they can reply to emails in the shared mailbox with the common From address. Now, one sales person might reply to an email (using the "sales" personality) and directly cc another person, who also happens to have a "sales" personality. If we just applied the "don't whitelist when sender matches any personality" rule, then those emails wouldn't be whitelisted when they should be.