When we speak of “Email Authentication”, we’re referring to a technique that is meant to provide to the recipient of a message some level of certainty that the message actually originated with the claimed source of the message. The idea is to achieve some level of defense against fraudulent email, such as phishing and spoofing, that could erode a recipient’s trust in receiving email. That said, the act of sending authenticated email does not, in and of itself, assert that the email is good or wanted email; it only means that the mail is such that a reputation for the authenticated party can be reliably established and used in email acceptance and placement decisions.

SPF Overview

Sender Policy Framework (SPF), to paraphrase RFC 7208, is a protocol that not only allows an organization to authorize hosts and networks to use its domain names when sending email, but also provides a way that a receiving host can check that authorization.

When you’re done reading this post, I hope you’ll have learned the following:

SPF is a “path-based” authentication system.

SPF policies are declared in DNS TXT records.

Validation is keyed to the Return-Path domain (what we here at SparkPost call the “bounce domain”) or the HELO domain.

Validation can be done early in the SMTP transaction, so it’s a good check to use to quickly reject unauthenticated mail.

It is prone to a relatively high percentage of false negatives.

“Path-Based” Authentication

SPF is deemed to be a path-based authentication system because it’s tied solely to the path the message took to get from its origin to its destination. When a sending server initiates an SMTP transaction with a receiving server, the receiving server will determine whether or not the sending server is authorized to send that message based on the domain’s SPF policy. It’s solely a decision based on where the message is coming from, with nothing whatsoever to do with the content of the message.

Declaring an SPF Policy

A domain’s SPF policy record is just a specifically formatted DNS TXT record, commonly containing one or more of the following “mechanisms”:

v=spf1 – Required first token to indicate that TXT record is SPF record (a domain can have multiple TXT records)

ipv4, ipv6 – Used to specify IP addresses and networks that are permitted to send mail for the domain

a – If the sending domain has a DNS “A” record that resolves to the sending IP, the IP is permitted

mx – If the sending IP is also one of the MX records for the sending domain, the IP is permitted

all – Required last token, always modified:

-all – Only what’s listed here can pass; reject failures.

~all – Stuff that’s not here should softfail; accept it but note it.

?all – Not sure if stuff that’s not here should pass.

+all – We think SPF is useless; pass everything.

Other less common mechanisms that are still in widespread use are:

include – Used when a domain not only has its own servers, but also uses someone else’s servers. Must be used judiciously. Each include is another DNS query, and SPF records can’t require more than ten DNS queries to resolve.

redirect – When domain A and domain B are owned by the same entity and use the same infrastructure. The owners typically want to maintain just one copy of the SPF record for both, and configure B to redirect queries to A.

exists – If a domain has lots of small, non-contiguous network blocks, it can use this mechanism to specify its SPF record, instead of listing them all. Useful to stay within size limit (512 bytes) for DNS response.

A note about the “ptr” Mechanism

A final mechanism, “ptr” existed in the original specification for SPF, but has been declared “do not use” in the current specification. The ptr mechanism was used to declare that if the sending IP address had a DNS PTR record that resolved to the domain name in question, then that IP address was authorized to send mail for the domain.

The current status of this mechanism is that it should not be used. However, sites doing SPF validation must accept it as valid.

Some Example SPF Records

Here are some example records, and their meanings. This example shows RFC 1918 addresses, but I recognize that such addresses will never show up in real SPF records.

MX record, one IP address, one IP network, softfail everything else:

“v=spf1 mx ipv4:10.0.0.1 ipv4:192.168.24.0/24 ~all”

A record, two IP networks, reject everything else:

“v=spf1 a ipv4:10.0.3.0/23 ipv4:192.168.55.0/26 -all”

We send no mail:

“v=spf1 -all”

We think SPF is stupid:

“v=spf1 +all”

The SPF record for the domain sparkpostmail.com (redirect mechanism used)

“v=spf1 redirect=_spf.sparkpostmail.com”

The SPF record for _spf.sparkpostmail.com (exists and ptr mechanisms used):

At SparkPost, we’re currently using the ptr mechanism in our SPF record. We have found it necessary due to a lack of universal support for validating the exists mechanism. And to date, we’ve seen no SPF failures result due to our using the ptr mechanism.

How Does SPF Authentication Work?

Here is what a typical SMTP transaction looks like when SPF is involved:

The sending server (client) connects to receiving server

The receiving server notes the client’s connecting IP address

They exchange greetings, including client’s HELO hostname

Client issues SMTP MAIL FROM command

The receiving server can now look up SPF record for either MAIL FROM domain or HELO hostname to determine whether or not the connecting IP is authorized to send mail for the domain, and decide whether or not to accept the message.

MAIL FROM or HELO – Which to Check?

The SPF specification stipulates that receiving sites MUST check SPF for the MAIL FROM domain, and RECOMMEND checking it for the HELO hostname. In practice, checking only happens on the MAIL FROM domain, except perhaps for those times when the MAIL FROM address is the NULL sender (i.e., “<>”), in which case the HELO hostname is the only identity available.

SPF Is Not Foolproof

While it seems a relatively straightforward way to authenticate email, SPF is vulnerable to failure in the form of false negatives. These failures can occur more frequently than many are comfortable with.

Here are two common scenarios where perfectly legitimate mail can fail SPF validation and so appear to be fraudulent:

Automated forwarding, where a message sent to jsmith@alumni.university.edu, a mailbox set to automatically forward all mail to jsmith@isp.com

Mailing lists, where mail sent to talk-about-stuff@listserv.foo.com gets “exploded” and sent to each subscriber

In both of these cases, if the Return-Path is unchanged from its original, the mail may fail SPF checks (depending on the modifier used with the ‘all’ mechanism). This is because it arrives at its final destination from an intermediate server, not the original one, and that intermediate server is not likely to be in the domain’s SPF record. A technique called “Sender Rewriting Scheme” or “SRS” can mitigate this risk, and some larger sites have adopted it, but it’s not universal.

Wrap Up

So that’s SPF authentication, how it works, how to declare an SPF policy, and what the pitfalls are in using SPF. SPF was the first email authentication scheme to achieve widespread adoption, but it’s not the only one out there. SPF authentication is most effective when deployed in combination with other anti-fraud techniques.

-Todd Herr

For further reading on email authentication, please see these great resources about topics like SPF, DKIM, and DMARC.

5 Comments

Dougon May. 3, 2017 at 7:47 am

Lots of people suggest SPF, but the downsides still seem unacceptable risk to me. Plus, my gut guess has been that most SPAM doesn’t bother with spoofing the envelope sender. Is that wrong? Is there any data available on how much SPAM uses which techniques?

You’re correct in your assessment that SPF has its shortcomings as an email authentication technique, especially when deployed alone. It was the industry’s first real attempt at codifying this sort of thing, and while it met with some success, the fact that DKIM and DMARC have both subsequently been advanced by the industry is a pretty strong indication to us that your thoughts are in alignment with the industry as a whole.

Here at SparkPost, we advise our customers to deploy all email authentication techniques available to them, which means publishing an SPF record (preferably ending in “-all”, if possible), signing all mail with DKIM, and publishing a DMARC record as well, with a policy of p=reject if at all possible. These three techniques taken in tandem are, we believe, the best way for a brand owner to reliably establish both his identity in email and an identity-based reputation with mailbox providers, while at the same time helping his customers maintain trust in the email they receive from that brand.

To the last part of your question, since the SparkPost platform does not handle inbound mail for our customers, we ourselves have no data on how much spam forges envelope senders or uses other tactics. Moreover, while a Google search for spam statistics turns up some hits, it’s not clear to us from where those sites are getting their data, nor how reliable it is, since there are wild variances in the numbers cited. We would suspect, too, that major mailbox providers would be relatively close-mouthed about what they’re seeing, so as not to alert the criminals as to how much the mailbox providers understand their techniques.

My main concern about SPF is the “false negatives”, specifically of any recipients who forward email to another destination. Is that no longer a significant issue? I guess SRS is supposed to address this, but is it prevalent enough now to negate the issue?

First, let’s agree that there are two kinds of forwarding – manual and automated.

In manual forwarding, party A receives an email, decides that party B should also read it, and uses his/her email client to forward the message to B. This kind of forwarding is not subject to SPF failure, because in this scenario, the Return-Path on the forwarded mail will be A’s email address, and it should transit the servers run by A’s mailbox provider.

In automated forwarding, party A has a rule in place to automatically forward all mail received to another address; a typical scenario here might involve the address smithj@alumni.foo.edu automatically forwarding to jsmith@foo.example.com. In this case, with party C sending mail to party A, the mail will arrive at foo.example.com with C’s address as the Return-Path, but with the connecting server being one belonging to alumni.foo.edu . This is the scenario that’s vulnerable to SPF failure, because it’s unlikely that C’s provider’s SPF record contains the server ranges for alumni.foo.edu .

Again, because we see very little inbound mail, we don’t have a good handle on how widespread such a problem might be. The two most obvious cases where an automated forward of a message would fail to be delivered would be:

C’s domain publishes only an SPF record, and that record ends in “-all”; in our experience, there aren’t many domains that do this
C’s domain publishes a DMARC record with a policy of p=reject, and the mail either isn’t DKIM signed or the DKIM signature fails to validate

There was quite an uproar in the industry roughly three years ago, when Yahoo published a p=reject policy with relatively little warning, and a significant volume of email that previously flowed freely stopped flowing. While the bigger hits in that case were to mailing lists and ESPs that allowed customers to use personal addresses as their sender address, automated forwarding took a hit then, too. As of now, it’s not something we at SparkPost hear much about, but we allow that it’s still a possibility under the above scenarios.

Dougon May. 5, 2017 at 2:57 pm

Yes, automated forwarding is what I’m concerned about. Anyone who uses a forwarding domain, for instance, as their “permanent” email address. (I’m aware of this issue because I personally did that for many years.)

Understand about you guys not having much data on that, though. I did a bunch of digging after this, and there doesn’t seem to be any clear answer as to how widespread this problem is today. SRS supposedly addresses it, and supposedly many providers use SRS now. I’m just trying to get a feel for what the chances are that if we enable it, some number of users will never receive our emails, and we’ll never know they didn’t.

DMARC: How to Protect Your Email Reputation

Infographic: How To Explain DKIM To Your Grandmother

So you're at dinner with Grandma enjoying some homemade apple pie when she pops the question. No, not when are you getting married, or when you are having kids, or when you are having grandkids, but what ...