Harvest notification emails are not going through to agency contacts

[adborden]

@hkdctol and other GSA contacts are receiving the emails, but agency contacts are not. Seems to have been happening for the past ~2 weeks.

[adborden]

@hkdctol sent me a copy of one of the messages. I didn't spot anything unusual with it. It passes SPF but is not signed with DKIM. Everything looks in order. Are the messages going to spam, or just not being delivered?

[adborden]

The only errors in the log are related to empty recipient addresses and they seem rare. So if there is an issue, it's between gsa's email server and the agency's email server. We could reach out to GSA to see if they've been seeing any issues sending our mail.

[hkdctol]

@adborden ok - let me check with agency contacts and then follow up with GSA email contacts.

[hkdctol]

This seems resolved--will close for now and reopen if we run into any agencies that are not receiving harvest reports through email.

[adborden]

Re-opening since we are getting reports from agencies that they are not receiving harvest notifications.

[adborden]

We updated our DMARC record to include a reporting address so that we can see and analyze delivery reports.

[adborden]

USAID says they haven't received an email since 3/25. Harvest jobs have been run everyday since then and I do see the email address in the mail.log on catalog-harvester1p

Mar 29 16:04:45 catalog-harvester1p postfix/smtp[24423]: 9CEC960584: to=<mdaniels@usaid.gov>, relay=smtp.gsa.gov[159.142.1.100]:25, delay=0.79, delays=0.01/0.01/0.62/0.16, dsn=2.0.0, status=sent (250 2.0.0 Ok: queued as DDAB721DE4)

Not seeing any errors in Agari, either.

[mogul]

@adborden is following up with USAID in a mail thread "Fwd: Email being marked as SPAM from data.gov".

[adborden]

I've added our USAID contact to the harvest job as a test and sent a test message directly from the harvest host (using mail). Moving to blocked until we hear back.

[mogul]

Gonna give this until Monday to see if they understand and can supply what we're asking for, then will follow up.

[adborden]

Heard from USAID, they are receiving emails but they are going to spam. Given our DMARC policy is reject, it seems unrelated to DMARC. They are still doing some investigation on their side, no action for us at this point.

[mogul]

@adborden will ping to see if there's any movement on the USAID side. However, we're going to close this issue since spam detection tuning on the receiving end is totally out of our control. If they come back with something specific we need to change, then we'll open a new issue.

[hkdctol]

Reopening. Heard from Energy, DOJ, and NEH yesterday that they have not received harvest reports since May 9. Same date for all. Checked harvest results from admin UI and the harvest reports have been going through. USAID also still not getting harvest emails.

Harvest report emails to internal gsa addresses being received as usual.

[hkdctol]

I don't think we've done any work since this ticket was reopened, but hearing from USAID and PBGC that daily harvest emails that they have not been getting for a while started coming through today. Waiting to hear back from other agencies.

[hkdctol]

DOJ got emails starting today too. Not sure we've ever really investigated anything specific. Maybe next step is asking GSA email team if something changed, since in this instance we have some consistent dates on when the problem started/stopped.

[hkdctol]

Seems like all the agencies are getting the emails. Reached out to a GSA email team member we consulted in the past to see if there's any explanation on the date range, May 9 - June 9.

[hkdctol]

All the agencies are getting the emails. We got some additional information from GSA email team, which we will relay to USAID.

[hkdctol]

Re-opening as I have multiple reports from agencies that the last harvest email received was September 7.

[adborden]

FYI, I've confirmed that we're handing off emails properly to the GSA SMTP relay. It looks like the emails are going through to GSA addresses, I received the DOJ report:

Oct  1 17:05:19 catalog-harvester1p postfix/smtpd[13327]: connect from localhost[127.0.0.1]
Oct  1 17:05:19 catalog-harvester1p postfix/smtpd[13327]: A4CB060344: client=localhost[127.0.0.1]
Oct  1 17:05:19 catalog-harvester1p postfix/cleanup[13330]: A4CB060344: message-id=<20201001170519.A4CB060344@catalog-harvester1p.prod-ocsit.bsp.gsa.gov>
Oct  1 17:05:19 catalog-harvester1p postfix/qmgr[5189]: A4CB060344: from=<no-reply@data.gov>, size=1771, nrcpt=4 (queue active)
Oct  1 17:05:19 catalog-harvester1p postfix/smtpd[13327]: disconnect from localhost[127.0.0.1]
Oct  1 17:05:20 catalog-harvester1p postfix/smtp[13331]: A4CB060344: to=<aaron.borden@gsa.gov>, relay=smtp.gsa.gov[159.142.67.242]:25, delay=0.64, delays=0.01/0.01/0.47/0.16, dsn
=2.0.0, status=sent (250 2.0.0 Ok: queued as 582BC29825)
Oct  1 17:05:20 catalog-harvester1p postfix/smtp[13331]: A4CB060344: to=<crystal.carter@gsa.gov>, relay=smtp.gsa.gov[159.142.67.242]:25, delay=0.64, delays=0.01/0.01/0.47/0.16, d
sn=2.0.0, status=sent (250 2.0.0 Ok: queued as 582BC29825)
Oct  1 17:05:20 catalog-harvester1p postfix/smtp[13331]: A4CB060344: to=<jake.bishopgreen@usdoj.gov>, relay=smtp.gsa.gov[159.142.67.242]:25, delay=0.64, delays=0.01/0.01/0.47/0.1
6, dsn=2.0.0, status=sent (250 2.0.0 Ok: queued as 582BC29825)
Oct  1 17:05:20 catalog-harvester1p postfix/smtp[13331]: A4CB060344: to=<monique.bourque@usdoj.gov>, relay=smtp.gsa.gov[159.142.67.242]:25, delay=0.64, delays=0.01/0.01/0.47/0.16
, dsn=2.0.0, status=sent (250 2.0.0 Ok: queued as 582BC29825)
Oct  1 17:05:20 catalog-harvester1p postfix/qmgr[5189]: A4CB060344: removed

We should open an incident with the GSA SMTP team.

[adborden]

I submitted a ticket with the email source.

[adborden]

Another report #907

[hkdctol]

Multiple agencies reporting they started getting the harvest report emails last night.

[mogul]

Leaving open until we see explicit confirmation of a change made on the GSA side that will prevent this improvement from regressing.

[adborden]

Here's the latest from GSA:

It looks like the Google SMTP server is changing the MAIL FROM to postmaster@gsa.gov for non-gsa.gov recipients, which is definitely wrong.

Using debug_peer_list = smtp.gsa.gov in the Postfix configuration, I was able to get a trace of the SMTP conversation on staging. This trace shows we are correctly setting the MAIL FROM as no-reply@data.gov on the handoff to GSA's SMTP server.

FWIW, I think the postmaster@gsa.gov issue is not new... we saw it with USAID but focused on a separate issue which seemed to resolve the delivery problems. Not sure if this is intermittent, or unrelated, or overlooked.

[adborden]

No response yet from GSA.

In the meantime, agencies are reporting that they are receiving harvest reports without any change on our side (or presumably GSA's).

Yet despite deliveries, I'm still seeing the From being replaced with postmaster@gsa.gov for non-gsa.gov recipients. Here you can see Google's mail servers connecting to mail.a14n.net, and the message being rejected with from=<postmaster@gsa.gov>:

Oct 15 00:49:56 a14n postfix/postscreen[7230]: CONNECT from [209.85.221.226]:33639 to [64.227.84.77]:25
Oct 15 00:50:02 a14n postfix/postscreen[7230]: PASS NEW [209.85.221.226]:33639
Oct 15 00:50:02 a14n postfix/smtpd[7231]: connect from mail-vk1-f226.google.com[209.85.221.226]
Oct 15 00:50:03 a14n postfix/smtpd[7231]: 4A5D313B0D9: client=mail-vk1-f226.google.com[209.85.221.226]
Oct 15 00:50:03 a14n postfix/cleanup[7257]: 4A5D313B0D9: message-id=<20201015004954.4E213BBE15@catalogharvester1d.dev-ocsit.bsp.gsa.gov>
Oct 15 00:50:06 a14n postfix/cleanup[7257]: 4A5D313B0D9: milter-reject: END-OF-MESSAGE from mail-vk1-f226.google.com[209.85.221.226]: 5.7.1 Spam message rejected; from=<postmaster@gsa.gov> to=<adborden@a14n.net> proto=ESMTP helo=<mail-vk1-f226.google.com>
Oct 15 00:50:06 a14n postfix/smtpd[7231]: disconnect from mail-vk1-f226.google.com[209.85.221.226] ehlo=2 starttls=1 mail=1 rcpt=1 data=0/1 quit=1 commands=6/7

Even with deliveries appearing resolved, we should press GSA on resolving the incorrect From address.

[mogul]

Leaving this Blocked while we wait on an explicit response from GSA IT.

[adborden]

The replacement of the From address to postmaster@gsa.gov, is part of an anti-spam feature of Google. Since data.gov is not registered as a GSA Google domain. This doesn't completely explain things, because while this was implemented, some mail is still going through.

That means that GSA SMTP service now requires any non-gsa.gov domains to be registered with GSA Google.

Long-term: we want to move away from the GSA SMTP service since it won't be able to support the cloud.gov use case where we are outside of the GSA network.

Short-term: GSA will add data.gsa.gov as a domain and we'll be able to send from no-reply@data.gsa.gov

[adborden]

GSA has opened a ticket for the new DNS entries.

[adborden]

The new sending domain is ready. I just tested it on datagov-jump2d and it seems to work. PR incoming...

[jbrown-xentity]

PR verified deployed on latest ckan production.ini file... smtp.mail_from = no-reploy@data.gsa.gov. Do not have a valid example in the wild.

[adborden]

Trying to keep a list of emails we send so that we know where the FROM address needs to be updated in the future