Antispam counter measures explained Part 1: How the Sender Policy Framework really works
Antispam counter measures explained Part 2: Advanced SPF records,  Best practice and biggest mistakes
Antispam counter measures explained Part 3: How DKIM really works and best practice

 

In Part 1 I explained the basics of SPF and how you can implement the most common scenario`s. After the basics we took a deep dive into how the Sender Policy Framework works and how it is  being evaluated. In this post I will take you even further than you have ever been inside SPF. If you are new to SPF I would advise you to first read the PART 1 blogpost. Let`s get on with the cool stuff.

Advanced SPF records:

In Part one I explained how mechanisms work, but there is 1 mechanisms that deviate from normal behavior. This mechanisms “include” is special because it can be used to manipulate your records and point to other records. Only this mechanisms does something different than you would expect it to do.  Further down the road we are going to talk about macro`s and modifiers. First thing first let`s take a closer look at the “include” mechanism.

“include”  mechanism:

When using a include mechanism the specified domain is searched for a match. If the include lookup returns a “match” its qualifier get called. If the include lookup returns a “Not match” processing proceeds to the next directive. If the include lookup returns an error the entire evaluation of the SPF record stops and the verdict is the error. Most mail receivers will reject based on a PermError or Temperror.

The qualifier of a match in a include:	Causes the “include” mechanism to:
Pass	match
Fail	not match
SoftFail	not match
Neutral	not match
TempError	throw TempError
PermError	throw PermError
None	throw PermError

The “include:” mechanism is meant to cross administrative boundaries. Great care is needed to ensure that “include:” mechanisms do not place domains at risk for giving SPF Pass results to messages that result from cross user forgery. Make sure technical mechanisms are in place at the specified other domain to prevent cross user forgery. The name “include” is poorly chosen. Only the evaluated result of the referenced SPF record is used, rather than acting as if the referenced SPF record was literally included in the first. For example, evaluating a “-all” directive in the referenced record does not terminate the overall processing and does not necessarily result in an overall Fail. (Better names for this mechanism would have been “if-pass”, “on-pass”, etc.)

Example DNS record:

tech-savvy.nl       TXT  “v=spf1 -include:spf.tech-savvy.nl +include:spf2.tech-savvy.nl -all”
spf.tech-savvy.nl   TXT  “v=spf1 +ip4:188.188.188.1 -all”
spf2.tech-savvy.nl  TXT  “v=spf1 +ip4:188.188.188.2 ~all”

If a email would come in from 2.2.2.2 using the spf as stated in the example the evaluation would be as followed:

1. The first include is being evaluated and it will search for a match.
   1. In the first include the first mechanism is evaluated. It will return a “no match”
   2. In the first include the second mechanism is evaluated. It has found a “match” and invokes the qualifier “-“. The verdict will be “No match” as “-” is not a Pass.
   3. the first include is ignored as no match has been found.
2. The second include is being evaluated and it will search for a match.
   1. In the second include the first mechanism is evaluated. It will return a “no match”
   2. In the second include the second mechanism is evaluated. It has found a “match” and invokes the qualifier “~”. The verdict will be “No match” as “~” is not a Pass.
   3. the second include is ignored as no match has been found.
3. The 3rd mechanism is evaluated and found a match on “all”. The qualifier “-” will be invoked and the verdict will be “SPF Fail”

Using includes to validate the sender by 2 or more mechanisms:

Wait a minute you explained that SPF will stop evaluation if a match is found and it will call its qualifier. So how can you do a double validation??.
If you take a good look at the qualifier table from part 1 you will see that there is a “+” qualifier. This means that we can apply reversed logic. Remember the hole meaning of SPF is getting a match back from the evaluation. The match can be the “+all” at the end of a SPF record and using multiple directives in the middle to filter  out unwanted senders with a “-” qualifier.

Example DNS records:

tech-savvy.nl               TXT    “v=spf1 -include:ip.tech-savvy.nl -include:email.tech-savvy.nl +all”
ip.tech-savvy.nl            TXT    “v=spf1 -ip4:188.188.188.1 +all”
email.tech-savvy.nl         TXT    “v=spf1 -exists:%{l}.tech-savvy.nl +all”
scriptkiddie.tech-savvy.nl  A

In the sample above the only mail that is getting a SPF pass is if the sending MTA is using IP “188.188.188.1 and the envelope from has “scriptkiddie” in the local part. This is due to the fact that a “-” qualifier in a include results in “No match”. The match has been made on the mechanisms but the qualifier it calls (“-“) reverts the evaluation to “Nomatch” causing further evaluation to halt and skip this include. So evaluation will continue with the next directive. Basically you need to fail both tests now in order to hit the SPF +all record in the root. Anything that hits a “+all” directive in the includes will result in a “Pass” causing the “-” qualifier on the include being called that produces the “No match” again.

Macro evaluations:

If you want to really fine tune what you are filtering out with your SPF policy and the default mechanism don`t offer you enough flexibility you can use macros. Macros are like variables or regex and can be used to dynamically define what should be checked.

The following macro letters are expanded in term arguments:

 

A literal “%” is expressed by “%%”.

 

Optional transformers are the following:

 

The following macro letters are allowed only in “exp” mechanism text:

 

If we look at the macro letters and transformers we can do some really cool stuff with macros. We know that a RBL ( Real Time Blocklist ) works based on IP lookups against its DNS. That is exactly what the mechanism “exists”  does. so we can now use “Exists” with a macro and offload RBL lookups directly int0 SPF preventing additional DNS load on your servers from doing RBL and SPF lookups.

example :

“v=spf1 -exists:%{ir}.zen.spamhaus.org include:servers.tech-savvy.nl include:others.tech-savvy.nl -all”

In this case the senders MTA IP address will be validated against the Spamhause ZEN database and if it is listed the verdict “Fail” will be called. For every spam server listed in ZEN database trying to deliver a mail, the receiving MTA will no longer do the include DNS lookups against your servers. This reduces potential DNS DDOS attacks with 66 % efficiency or more depending on the remaining records.

Some more macro examples:

• The <sender> is Iam-a-badsender@my.domain.com.
• The IPv4 SMTP client IP is 188.177.166.155
• The IPv6 SMTP client IP is 2001:DB8::CB01
• The PTR domain name of the client IP is test.domain.com.

macro                       expansion
-------  ----------------------------
%{s}    Iam-a-badsender@my.domain.com
%{o}                    my.domain.com
%{d}                    my.domain.com
%{d2}                      domain.com
%{d1}                             com
%{dr}                   com.domain.my
%{l}                  Iam-a-badsender
%{l-}                 Iam.a.badsender
%{lr}                 badsender-a-Iam

macro-string                                               expansion
--------------------------------------------------------------------
%{ir}.%{v}._spf.%{d2}        155.166.177.188.in-addr._spf.domain.com
%{lr-}.lp._spf.%{d1}                     badsender-a-Iam.lp._spf.com

%{lr-}.lp.%{ir}.%{v}._spf.%{d2}
                    bad.strong.lp.3.2.0.192.in-addr._spf.example.com
%{d}.trusted-domains.tech-savvy.nl
                         my.domain.com.trusted-domains.tech-savvy.nl

IPv6:
%{ir}.%{v}._spf.%{d2}                               1.0.B.C.0.0.0.0.
0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.8.B.D.0.1.0.0.2.ip6._spf.domain.com

SPF Modifiers:

Modifiers are optional and can be used to redirect or provide additional error information back to the sender. Currently there are only 2 modifiers, wile more where planned they never made it to the production RFC.

The “redirect” modifier:

If you have lots of subdomains and all of them would need to have the same SPF record as your root domain, it is best to use the redirect modifier. This way you can better manage the administrative overhead when a change is needed. The redirect modifier will cause the evaluation of the SPF record to use the domains SPF record that is mentioned after the “=”.  The macro-expanded domain is also substituted for the current-domain in those look-ups. In the following example, the client IP is 1.2.3.4 and the current-domain is child.tech-savvy.nl

DNS Example:
child.tech-savvy.nl TXT  “v=spf1 redirect=tech-savvy.nl”
tech-savvy.nl       TXT  “v=spf1 ip:1.2.3.4 -all”

1. If tech-savvy.nl has no SPF record, the result is error
2. Look up the SPF record for tech-savvy.nl.
3. validate if it matches 1.2.3.4
4. return Pass
5. If there is no match, the exec fails to match, and the -all value is used.

You should never use a “all” mechanism when using a redirect.

The “exp” modifier:

Every body knows that service ticket that’s is coming in when a customer says: “My mail did not get delivered here is the NDR” (At least if your lucky the end user send the NDR to you). When you look at the NDR or the DSN code your brain goes like “That code is familiar but what was it” , so you hit google to search for that specific DSN code. What if you could have the receiving mail server send a custom error about this specific SPF failure inside the NDR. Well that is what the “exp” modifier is all about. When a SPF evaluation fails it will search for a “exp” modifier and if found evaluate it. The “exp” tag itself always points to a txt record and in that txt record you can use the macros mentioned above to create custom reports that will be send back in the NDR.

Example DNS records:

tech-savvy.nl           TXT  “v=spf1 ip:188.188.188.188 -all exp:explain.tech-savvy.nl”
explain.tech-savvy.nl   TXT  “SPF error %{i} is not one of %{d}’s designated mail servers.”

or have a website setup that can accept parameters for customized help:

tech-savvy.nl          TXT   “v=spf1 ip:188.188.188.188 -all exp:explain.tech-savvy.nl”
explain.tech-savvy.nl  TXT   “SPF error See http://%{d}/why.html?s=%{S}&i=%{I}”

Best Practices and biggest mistakes:

If you would read the RFC of SPF and DMARC you will find a lot of mismatches and best practice conflicts. I will try to give you some advise into what you can do and should not do, but there is no exact guideline. Due to these deviations MTA’s have different implementations of how they handle the verdicts and how they or there owners handle error messages. Just like with all spam counter measures the receiver decides if he honors the verdict and your policy. I will share some advice on what I think you can consider as a best practices and don’t do’s.

Optimization advise:

• Make use of the SPF sequence lookup. Set your most used servers at the front of the SPF record.
  Example: You have multiple domains and use includes for all subdomains. You have a on premises relay and office 365. O365 is responsible for sending out 80% of your mail while your relay is only responsible for 20% of outbound mail. If your SPF record would be “v=spf1 include:spf.myrootdomain.nl include:spf.protection.outlook.com ~all” All traffic originating from O365 will hit your DNS servers unnecessary. You should use “v=spf1 include:spf.protection.outlook.com include:spf.myrootdomain.nl  ~all”. This way you save 50% DNS hits and even 75% or more if there are DNS entry’s in your “spf.myrootdomain.nl”  zone.
• Split your own servers from 3rd party servers by using “include”. This way other party’s can include your servers without including the 3rd party’s you have included. In the example below you can have other domains (Where you send mail on their behalf) use your record “myservers.tech-savvy.nl” as a include. This way they will not authorise the 3rd party (servers.mcsv.net)  servers in their domain.

Example DNS record:

tech-savvy.nl             TXT   “v=spf1 include:myservers.tech-savvy.nl include:3rdparty.tech-savvy.nl -all”
myservers.tech-savvy.nl   TXT   “v=spf1 +ip4:188.188.188.1 -all”
3rdparty.tech-savvy.nl    TXT   “v=spf1 include:servers.mcsv.net -all”

• Use redirect and include when your servers are responsible for many (sub)domains.

No brainers:

• List a server only Once, having servers listed multiple times in different includes only bloats your record. remember SPF evaluation stops at the first match.
• Only list outbound servers that send mail out.
• Test your SPF records before you implement them. ( See part 1 for record testers )
• Make sure the MTA that is doing the SPF check is not behind a NAT device.
• Inform your marketing department and application departments. Specially marketing tents to host advertising campaigns via 3rd party mass mail solutions.

SPF <-> DMARC conflicts:

• SPF best practice is to use “?” qualifier for includes that are pointing to 3rd party’s. DMARC recognizes “?” qualifier as a “Fail” and will quarantine or drop your mail.

Increased protection:

• Know what is happening with your domain. Use DMARC reporting to get inside in who is sending on your behalf.
  There has been a SPF reporting RFC (RFC6652) but it has never reached the implementation status. DMARC Reporting will be explain in a separate post.
• Harden your security by combining SPF with DKIM.
  With DKIM you can protect email in transit this will be further explained I a separate post.
• Publish null SPF records for your domains that don’t send mail. “v=spf1 -all”

Common mistakes and Dont’s:

• Use a “all” mechanism with a redirect modifier.
• Many administrator think they have protected their domain when they use “~all”.
• Do not use “Include” and have it pointed to an empty Record.
• Do not change you record to use “include” or “redirect” when the record it points to does not yet exist. First create / change the record in the include, wait till TLL has passed and then change the “include” or “redirect” to the record that needed them. If you have a 1 day TTL you can lose all mail from 1 day if you change it in the wrong order.
• Do not enable SPF check on a MTA that is not on your edge mail infrastructure.

If you are even more interested you can read the entire RFC for SPF at https://www.ietf.org/rfc/rfc4408.txt

This is the end of Part 2 regarding SPF. Stay tuned for part 3 where we will dive a little deeper into DKIM.

Martijn van Geffen