Reprinted from TidBITS by permission; reuse governed by Creative Commons license BY-NC-ND 3.0. TidBITS has offered years of thoughtful commentary on Apple and Internet topics. For free email subscriptions and access to the entire TidBITS archive, visit http://www.tidbits.com/ Yahoo and AOL Damage Mailing Lists and Email Forwarding Adam C. Engst I recently received an email message from Cornell Alumni Affairs, [1]warning of deliverability problems related to alumni email forwarding and participation in Cornell mailing lists. Since I've recently spent quite a bit of time learning about SPF, DKIM, DMARC, and other email deliverability technologies for TidBITS and Take Control, this piqued my interest. Some additional investigation showed that the problem extends far beyond Cornell, and is currently affecting any organization that relies on email forwarding, plus any mailing list with Yahoo or AOL subscribers. [2][tn_Cornell-DMARC-email.jpg] First, to explain all this, we need to untangle the alphabet soup of technologies I blithely splashed into the above paragraph, all of which are designed to reduce the ability of spammers to spoof email to make it look like it comes from a legitimate sender. * [3]SPF stands for Sender Policy Framework and enables the owner of an Internet domain, like tidbits.com, to specify which computers are allowed to send email containing sender addresses in that domain. A mail server receiving email from a sender claiming to be foobar@tidbits.com does a DNS lookup on tidbits.com, checking a special TXT record to see if the sender's mail server is allowed to send email on behalf of tidbits.com. You can think of SPF working at the envelope level, and it's an Internet Engineering Task Force standard ([4]RFC 7208). * [5]DKIM stands for DomainKeys Identified Mail, and is also an IETF standard ([6]RFC 6376). Whereas SPF looks at the envelope to verify that a message is being sent from an approved source, DKIM goes one step deeper, associating a domain name with an actual email message, via public key cryptography. First, the sending mail server uses a private key to sign the contents of each outgoing message using a DKIM-Signature header. Second, the receiving mail server does a DNS lookup on the domain name specified in the DKIM-Signature header to find another special TXT record that contains the associated public key; the receiving mail server can then use the public key to verify that the message hasn't changed since it was signed. * [7]DMARC, or Domain-based Message Authentication, Reporting, and Conformance, is an email authentication method that builds on SPF and DKIM. Again implemented in DNS, a DMARC policy lets a sending organization specify that its messages use SPF and/or DKIM, and what should happen to messages for which the domain in the header's From line fails to match (is not in 'alignment' with) the domains specified by SPF and DKIM. There are three options: do nothing ('none'), flag the message as suspicious ('quarantine'), or bounce the message ('reject'). Although DMARC is used by numerous major email providers in various ways, [8]it is not an IETF standard. A key reason DMARC exists is that the ways SPF and DKIM enable the sender to specify what should be done with failed messages are generally unused. In SPF, each receiving mail server has to determine what to do with results such as none, neutral, pass, fail, and softfail. Although fail was intended to equate to rejection, few senders set that, due to the high number of false positives. The DKIM specification is even more explicit about how its results should (not) be used: In general, modules that consume DKIM verification output SHOULD NOT determine message acceptability based solely on a lack of any signature or on an unverifiable signature; such rejection would cause severe interoperability problems. (There is a little-used extension to DKIM called [9]ADSP (Author Domain Signing Practices, [10]RFC 5617), which was the predecessor to DMARC and offers advice from the sender when the From header doesn't match the domain in the DKIM signature. It was quickly identified as having the same problems that now afflict DMARC; hence its minimal usage.) So DMARC allows a sender to tell receiving mail servers what to do with messages whose whose SPF- and DKIM-advertised domains don't match the domain in the From line. That's ideal for companies like PayPal, which send vast quantities of transactional email, are constantly spoofed as part of phishing attacks, and don't have (many) individual users. PayPal has published a DMARC policy of 'reject' for over a year with no one noticing. However, until recently, large email providers have stuck with a DMARC policy of 'none,' which lets messages be delivered normally. You can check the DMARC policy for any domain at the [11]DMARC Inspector ' look at the 'p=' tag. In April 2014, [12]Yahoo dropped a bomb on the email community by quietly changing its DMARC policy to 'reject,' and [13]AOL followed suit shortly after, though at least with [14]a press release acknowledging the change. This had the effect of causing receiving mail servers to bounce messages that failed SPF and DKIM, generating two classes of problems: * Email Forwarding: Let's say you have a cornell.edu alumni address, which forwards to a Gmail address. Someone from Yahoo or AOL sends you a message, which uses both SPF and DKIM. Receiving it at cornell.edu works fine, because SPF, DKIM, and alignment all pass at that point, but when Cornell's mail server forwards the message, it rewrites headers such that, when Gmail examines the new incoming message, it fails SPF, DKIM, and/or alignment. Because Yahoo and AOL both have DMARC policies saying that failed messages should be rejected, Gmail bounces the message. * Mailing Lists: Now let's say you use Yahoo or AOL and are a member of a discussion-based mailing list. When you post to the list, the list server receives the message and packages it for distribution to the list, changing a variety of headers and possibly adding a signature to the body in the process. Those changes cause the repackaged message to fail SPF, DKIM, and, most important, alignment, so it won't be received by any list recipients at Yahoo, AOL, Gmail, Outlook.com, or other any other email provider that honors DMARC policies, again because of how Yahoo and AOL set their DMARC policies to require that failed messages be rejected. There's even more fallout on mailing lists. Depending how things are configured, if Gmail bounces a received mailing list message sent by a Yahoo user, that bounce will likely go back to the mailing list server and be recorded against the Gmail user, potentially causing that user to be removed from the list. So Yahoo and AOL users can get other list members bounced, purely by posting. Needless to say, as email and mailing list administrators have figured out what is going on, this change has caused significant consternation. The [15]IETF discussion list has been dominated by DMARC-related discussions for months. Why did Yahoo and AOL make such a sweeping change? Speculation is that both had suffered significant security breaches that allowed bad guys to steal user information, including address books. They then used that information to create spam to these users' contacts, forging the users' return addresses. Since that spam was being sent by botnets, nothing short of this DMARC change could stop it. In short, Yahoo and AOL are cleaning up their mistakes by damaging every mail forwarding service and mailing list on the Internet. What can be done about this situation? Although email boffin John Levine has compiled [16]a list of things mailing list developers and administrators can theoretically do, none are trivial and all have side effects. At the user level, there's simply nothing that can be done, other than those who use Yahoo and AOL voting with their feet and switching to less-draconian free email providers like iCloud, Gmail, and Outlook.com. Or, if you want to pay an email provider to ensure that they're working for you, TidBITS staffers have used [17]FastMail and easyDNS's [18]easyMail service (which comes with domain hosting). As an aside, although iCloud doesn't appear to be involved with these DMARC-related problems, [19]beware of iCloud's spam filtering. In the most recent case, Doug Adams, who has contributed hugely to the Apple community through his [20]Doug's AppleScripts for iTunes site, found that for three weeks his messages weren't being received by iCloud users because [21]Apple was silently deleting all messages that contained the string 'dougscripts'. Since his domain is dougscripts.com, that pretty much put the kibosh on all his iCloud-related email. After complaining to AppleCare and having the problem escalated to a Mac Advisor, and then to Engineering, and then to the Spam Team, the problem finally disappeared, without further explanation. It's astonishing that, in 2014, Apple is still doing simple string-based filtering and silent deletion of offending messages. What I'd like you to take away from this article, apart from just understanding the Internet a bit better, is that there's a lot of effort that goes on behind the scenes to make sure even the most basic of Internet technologies work in the face of competing forces. Email may seem simple, but it's also far moresubtle than most people realize. References 1. http://www.alumni.cornell.edu/services/dmarcissue.cfm 2. http://tidbits.com/resources/2014-06/Cornell-DMARC-email.png 3. http://en.wikipedia.org/wiki/Sender_Policy_Framework 4. http://tools.ietf.org/html/rfc7208 5. http://en.wikipedia.org/wiki/DomainKeys_Identified_Mail 6. http://tools.ietf.org/html/rfc6376 7. http://en.wikipedia.org/wiki/DMARC 8. http://dmarc.org/ 9. http://en.wikipedia.org/wiki/Author_Domain_Signing_Practices 10. http://tools.ietf.org/html/rfc5617 11. https://dmarcian.com/dmarc-inspector/yahoo.com 12. http://jl.ly/Email/yahoobomb.html 13. http://jl.ly/Email/aoldmarc.html 14. http://postmaster-blog.aol.com/2014/04/22/aol-mail-updates-dmarc-policy-to-reject/ 15. http://www.ietf.org/mail-archive/web/ietf/current/threads.html 16. http://wiki.asrg.sp.am/wiki/Mitigating_DMARC_damage_to_third_party_mail 17. https://www.fastmail.fm/ 18. https://web.easydns.com/DNS_Hosting_and_Domain_Management_Services/ 19. http://www.macworld.com/article/2029570/silent-email-filtering-makes-icloud-an-unreliable-option.html 20. http://dougscripts.com/ 21. http://dougscripts.com/itunes/2014/06/the-silent-filtering-treatment/ .