Access Keys:
Skip to content (Access Key - 0)

Email Delivery - Underlying Protocols

On this page:

Overview

Sending and receiving email, though it seems like it "just works," is a very complex process on the back-end with many technologies and protocols in play. There are many reasons articles may be identified as spam and not delivered to your inbox. This article aims to explain the three main technologies/protocols (SPF, DKIM, and DMARC) that influence if an email message will be delivered to an inbox, or if the email message will be quarantined by a spam filtering service.

Traditional Technologies/Protocols

Sender Policy Framework (SPF): Allows a domain owner to specify which IP addresses are authorized to send an email on their behalf.

DomainKeys Identified Mail (DKIM): Uses an encrypted signature to verify if an email sender is who they say they are, and gives a key to the recipient to verify this with the sender's DNS record. A valid DKIM signature verifies that some parts of the email have not been modified since the signature was affixed.

Domain-based Message Authentication Reporting & Conformance (DMARC): Open protocol to prevent phishing attacks via impersonation (also known as "spoofing"). DMARC combines the results from SPF and DKIM to accurately identify if an email is from either an authorized sender or a fraudulent impersonator, and can therefore actively block phishing attacks by enforcing a policy that only allows emails from authorized senders to be delivered. DMARC can instruct receiving servers on how to handle inbound emails.  Email servers have the option to enforce one of the following DMARC policies:

  1. None - Accept all email
  2. Quarantine - Send email that fails validation to spam
  3. Reject - Actively reject all unauthorized email

Sender Rewriting Scheme (SRS): SRS is a technique for rewriting the sender of an email message when it’s forwarded across domains, and was developed to address the forwarding issues introduced by SPF, where forwarded messages will appear to not comply with the list of approved IP addresses for email to be sourced from for a given domain.
MIT has supported SRS for alum.mit.edu email for life accounts since fall of 2015. In late December of 2018, MIT implemented this functionality for MIT Exchange email accounts to address reports of failed forwarding to Google and other providers due to SPF failures.

MIT Email Forwarding

Forwarding email, in general, causes a lot of DMARC failures.

In instances where DKIM is breaking when a message is encoded, one of the components of DMARC will then fail when the message is delivered to the external provider (because the signature is no longer verifiable). This results in high spam scores and messages being identified as spam and quarantined or not delivered by the external provider.

This kind of delivery failure has been known to happen with organizations such as the NSF (National Science Foundation), causing messages from them sent to MIT community members who use forwarding to not receive important communications. If you're in regular communication with any such organizations, we do not recommend using MIT's email forwarding service.

Non-delivery receipts also can be marked as spam in error due to their unique routing paths. These messages traverse the on-premises Exchange mail system (for users with Exchange mailboxes on-premises). That can cause DMARC to fail and the message to be identified as spam when it is delivered to the external provider.

Mail Forwarding Examples

Yahoo Sets Their Policy
Yahoo publishes a policy that all mail from their domain needs to conform with their published DMARC policy or otherwise be rejected.

Non-Forwarded Mail

  1. Mail is sent to your MIT email address from a yahoo.com address.
  2. Microsoft's O365 mail servers receive the message on MIT's behalf from Yahoo's servers.
  3. Microsoft's O365 mail servers look up the DMARC record for yahoo.com and determines the message came from one of yahoo.com's verified servers and that the DKIM signature validates correctly.
  4. The message passes validation.

Forwarded Mail

  1. Mail is sent to your MIT email address from a yahoo.com address.
  2. Microsoft's O365 mail servers receive the message on MIT's behalf from yahoo's servers.
  3. The message is delivered from Microsoft O365 to MIT's on-premises Microsoft Exchange email system.
  4. Microsoft Exchange stores the message in its internal database.
  5. Microsoft Exchange reassembles the message and forwards it to the user's configured forwarding address. As part of this process, the DKIM signature is broken.
  6. Your external provider looks up the DMARC record for yahoo.com and determines that all messages from that domain must comply with Yahoo's DMARC policy or be rejected.
  7. The message does not pass DKIM validation due to the extraction / reassembly process.
  8. Per yahoo.com's rules, the message is rejected as spam.

For additional detail, please see Microsoft's post on this subject: Why does my email from Facebook, that I forward from my outlook.com account, get rejected?

As noted in Microsoft's article above, changes have been made to O365's Exchange implementation to address this issue and avoid disturbing DKIM signatures.

Mailing Lists and DMARC

While DMARC is helpful in reducing spam and phishing, it has negative consequences for mailing lists. These consequences apply to Mailman, Listserv, etc., but not Moira lists. For example, Yahoo! published a DMARC policy that rejects all messages that fail DMARC signature tests several years ago, so affected mailing lists with Yahoo! members started seeing bounces from these members after that policy was published. There are no easy solutions to these consequences, and most involve changing what were previously standard mailing list practices. For more information, see:

Quarantined Message Header Examples

SPF Failure

The SPF failed because the sender IP did not match one of the ones the domain owner identified as a verified sender.

Authentication-Results: spf=fail (sender IP is 18.9.1.45)
 smtp.mailfrom=ufrpe.br; mitprod.mail.onmicrosoft.com; dkim=pass (signature
 was verified) header.d=mit.edu;mitprod.mail.onmicrosoft.com; dmarc=none
 action=none header.from=ufrpe.br;
Received-SPF: Fail (protection.outlook.com: domain of ufrpe.br does not
 designate 18.9.1.45 as permitted sender) receiver=protection.outlook.com;
 client-ip=18.9.1.45; helo=hybrid.exchange.mit.edu;

DMARC Failure

While the SPF passed, the DKIM is neutral because the body hash failed. That's why the DMARC failed.

Received: from NAM04-SN1-obe.outbound.protection.outlook.com
(mail-eopbgr700121.outbound.protection.outlook.com. [40.107.70.121])
        by mx.google.com with ESMTPS id g131si20669474wmf.32.2018.12.29.10.29.24
        for <mit@joeuser.com>
        (version=TLS1_2 cipher=ECDHE-RSA-AES128-SHA bits=128/128);
        Sat, 29 Dec 2018 10:29:24 -0800 (PST)
Received-SPF: pass (google.com: domain of joeuser@mit.edu designates
40.107.70.121 as permitted sender) client-ip=40.107.70.121;
Authentication-Results: mx.google.com;
       dkim=pass header.i=@mit.edu header.s=selector1 header.b=dTCWjU4o;
       dkim=neutral (body hash did not verify) header.i=@twitter.com
header.s=dkim-201406 header.b=mTUtkyGh;
       spf=pass (google.com: domain of joeuser@mit.edu designates
40.107.70.121 as permitted sender) smtp.mailfrom=joeuser@mit.edu;
       dmarc=fail (p=REJECT sp=REJECT dis=QUARANTINE) header.from=twitter.com

Analyze Your Headers

Microsoft offers a Header Analyzer that coupled with their glossary can help you understand the headers of any of your messages. Simply input the header into the analyzer and click on the headers to open the glossary and learn more about what each means.

Key headers related to spam identification include:

  • Received-SPF: SPF score
  • Authentication-Results: Spam/protocol scores
  • ARC-Authentication-Results: Spam/protocol scores
  • PCL: Phishing Confidence Level
  • SCL: Spam Confidence Level
  1. Login to your quarantine at https://protection.office.com/.
  2. *Threat Management > Review > Quarantine *
  3. Select the message > View message header
  4. Click Copy message header > Microsoft Message Header Analyzer
  5. Paste header into Analyzer and click Analyze Headers
  6. Click on a header to open the definition of it in the glossary.



ARC - The emerging authentication standard

Authenticated Received Chain (ARC): A standard created in 2016 that aims to fix instances where SPF and/or DKIM may break as a forwarded email goes through various hops during delivery. Without ARC, when each recipient system receives a forwarded message, the recipient system pays no attention to any credentials except for those from the most recent hop. In many of these cases, the "hop" had to re-sign the message that it is relaying, and in the process of re-signing the message, it "breaks" the original digital signature from the initial sender (as is the case with the current MIT email environment). Since the DKIM signature is now broken, when the email arrives at the new recipient system, it will fail DMARC. With ARC, any forwarding entities assume a level of additional "trust" when they are merely relaying the message by adding a "seal" and a "message signature." This allows the original email, along with its corresponding SPF and DKIM to remain intact. The forwarding entity basically says "I haven't changed anything, I'm just the go-between, and here are my credentials so that you know that I am who I claim to be."

While some email providers have already moved to ARC, Microsoft has yet to implement the standard. When it is implemented, users who forward their MIT email will see less instances of mail being mis-flagged as spam by external email providers.

See Also

IS&T Contributions

Documentation and information provided by IS&T staff members


Last Modified:

June 21, 2019

Get Help

Request help
from the Help Desk
Report a security incident
to the Security Team
Labels:
c-spam-filtering c-spam-filtering Delete
spam spam Delete
protocol protocol Delete
email email Delete
Enter labels to add to this page:
Please wait 
Looking for a label? Just start typing.
Feedback
This product/service is:
Easy to use
Average
Difficult to use

This article is:
Helpful
Inaccurate
Obsolete
Adaptavist Theme Builder (4.2.3) Powered by Atlassian Confluence 3.5.13, the Enterprise Wiki