Check/update SendGrid config to make sure we're meeting new email requirements

[becky-gilbert]

TL;DR

According to SendGrid, Gmail and Yahoo are enforcing new requirements for what emails they'll accept and send to users, so we need to make sure that we're meeting these requirements before they go into effect (Feb 1st).

Narrative

If we don't meet the new requirements that are being enforced by Gmail and Yahoo, any of our users (families and researchers) who are using email accounts with these providers will not receive our automated emails. SendGrid has a blog post about this here: https://sendgrid.com/en-us/blog/gmail-yahoo-sender-requirements.

We'll need to go through each of the items listed in the blog post and check to see if we meet the requirement. Depending on whether we're failing any of these requirements and how much work they will be to change/fix, we may need to create separate issues for the individual items (below).

See also Google's email sender guidelines: https://support.google.com/a/answer/81126?visit_id=01706211467783-8090904268743188225&rd=1

Items to check/fix

I have ordered these items roughly in order of priority based on (1) how easy it is to do and (2) whether or how likely it is to affect us.

• Set up DMARC email authentication for your sending domain. To do this, we need to set up SPF and DKIM email authentication. - This requires action from MIT IS&T. I've opened a ticket. - @becky-gilbert
• Ensure that sending domains or IPs have valid forward and reverse DNS records, also referred to as PTR records. - This requires action from MIT IS&T. I've opened a ticket. - @becky-gilbert
• For subscribed messages, enable one-click unsubscribe and include a clearly visible unsubscribe link in the message body. - SendGrid does offer a one-click unsubscribe service that we might want to use. It would add the 'unsubscribe' buttons to our emails, which would trigger an email and/or http POST request, which we would need to handle. - @okaycj One-Click Unsubscribe #1350
• Don’t impersonate Gmail From: headers. And the domain in the sender's From: header must be aligned with either the SPF domain or the DKIM domain. - We currently send emails from mit.edu, as well as from lab emails (which could include gmail). The use of lab email addresses as the "From" address might be a problem. - @okaycj Fix 'from' headers from labs, and edit custom email template #1356
• Use a TLS connection for transmitting email.
• Format messages according to the Internet Message Format standard (RFC 5322). - I think we're already doing this but need to check.
• Keep spam rates reported in Postmaster Tools below 0.3%. (We may need to set up Google Postmaster too). - I don't think we have Google Postmaster set up. SendGrid dashboard indicates our spam rates are below 0.3%.
• If you regularly forward email (including using mailing lists or inbound gateways), add ARC headers to outgoing email. - Does not apply to us.

Deadline/timeline

From the SendGrid blog post:

If a sender does not meet the requirements by February 2024, they will start to see temporary errors occurring on a small percentage of their non-compliant mail to Google recipients. In April 2024, a small percentage of the mail will be rejected and that percentage will gradually increase over time. The requirement for senders to implement one click unsubscribe will not be enforced until June 2024.

Update 2/6/24

We are getting a number of failed emails with this message:

550 5.7.26 Unauthenticated email from is not accepted due to domain's DMARC policy. Please contact the administrator of domain if this was a legitimate mail. To learn about the DMARC initiative, go to https://support.google.com/mail/?p=DmarcRejection l129-20020a257087000000b00dc6c172e33dsi1600971ybc.707 - gsmtp

Prior to 2/1/24, there were already some domains producing this error:

• berkeley.edu
• asu.edu
• fiu.edu
• ucsd.edu

After 2/1/24, we are seeing additional 'unauthenticated' errors from emails with these domains:

• bc.edu
• gmail.com

It looks like there've been approx 70 such errors from 2/1-2/6/24, and a similar number in all of January.

Related existing issues

• Add opt-out info to study_announcement email templates #673
• Improve deliverability of email #674

Other issues to consider addressing now

• Avoid hard-coding links and email addresses #670
• Change links in recruitment emails from Lookit to CHS #1336 - in progress via Make SECONDARY_DOMAIN redirect to SITE_DOMAIN lookit-orchestrator#49
• Use study-level parameters for participant compensation/communication to improve family and researcher communication #1314

There are a number of other open email-related issues, but because of the deadline for addressing the critical issues here, I think they should all be considered out-of-scope.

[ianchandlercampbell]

@becky-gilbert I just re-read the blogpost and I think we have a longer on-ramp to deal with these issues than we were thinking: https://sendgrid.com/en-us/blog/gmail-yahoo-sender-requirements.

"In a new effort to further protect their users’ inboxes, both [Gmail](https://blog.google/products/gmail/gmail-security-authentication-
spam-protection/) and Yahoo! introduced a new set of requirements senders must meet by February 2024 in order for mail to be delivered as expected to their subscribers. If a sender does not meet the requirements by February 2024, they will start to see temporary errors occurring on a small percentage of their non-compliant mail to Google recipients. In April 2024, a small percentage of the mail will be rejected and that percentage will gradually increase over time. The requirement for senders to implement one click unsubscribe will not be enforced until June 2024".

I have emailed Jason to get a meeting so we stay ahead of this, but it looks like no apocalypse scenario in February!

[becky-gilbert]

@ianchandlercampbell awesome, thanks for the update!

[becky-gilbert]

Update 3/27/24

The following items from the checklist above are now in production via #1366:

• For subscribed messages, enable one-click unsubscribe and include a clearly visible unsubscribe link in the message body. One-Click Unsubscribe #1350
• Don’t impersonate Gmail From: headers. And the domain in the sender's From: header must be aligned with either the SPF domain or the DKIM domain. Fix 'from' headers from labs, and edit custom email template #1356

[becky-gilbert]

I'm not seeing any DMARC or other sender-related delivery errors in SendGrid since these changes went into production, so I'm closing this issue as completed in the last production release (#1366). We should continue to monitor email delivery in SendGrid, and we can reopen this if any errors pop back up.

[ianchandlercampbell]

Huzzah! Sounds good Ian Chandler-Campbell, Ph.D. Open-Source Community Manager, Children Helping Science <https://childrenhelpingscience.com/> Research Associate, Project GARDEN <https://childrenhelpingscience.com/garden/scientists> ianchandlercampbell.com he/him/his

…

On Thu, Apr 4, 2024 at 1:10 PM Becky Gilbert ***@***.***> wrote: I'm not seeing any DMARC or other sender-related delivery errors in SendGrid since these changes went into production, so I'm closing this issue as completed in the last production release (#1366 <#1366>). We should continue to monitor email delivery in SendGrid, and we can reopen this if any errors pop back up. — Reply to this email directly, view it on GitHub <#1340 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ASABGQ23XEKBWVEUVFE5SMTY3WJSXAVCNFSM6AAAAABCA3FJTSVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDAMZXHA3TGMRWGA> . You are receiving this because you were mentioned.Message ID: ***@***.***>