Email

[Alex-Jordan]

I sat down to revise how we do email, but I realized a decision needs to be made by the community. [Somewhere in the past few months there was a thread about this, but I cannot find it.]

The issues around this are off-radar for typical installations. A "typical installation" is at an institution where the WW server uses a subdomain of the institution, and uses the institution's own mail server.

The issues are:

• Even for a "typical installation", if a user changes their email address to something that does not use the institutional domain, then they send an email that is "from" that non-institutional email address. And then the institutional mail server may be configured to refuse to actually send the message. It will bounce it back to the WW server. This would be a policy to protect the institutional mail server from being blacklisted as a spammer.
• For a non-typical installation, what is it using as a mail server? Let's say it uses localhost with postfix. Now if users have an institutional email address, your localhost sends out messages that are "from" person@school.edu. But of course the message can't be signed as actually coming from school.edu. Recipient mail servers will see this and it's a red flag for spam detection. Recently Google and Yahoo hardened their policies about this, so if you were hoping to send to an instructor at a school that uses Gmail under the hood, the message could be rejected (not even making it to the instructor's spam folder).
• We don't have the infrastructure to work with a paid mail service, like say Google. If we had this, one option for a non-typical installation would be to offload the painfully tedious details of mail management over to a service like that.

With the RS server, I'm in that second category. I changed how emails go out. If Stu Student uses the Email Instructor button, the reply-to address in the email is still Stu's actual email address. But the "from" address is something like "Stu Student <stu.student@webwork-hosting.runestone.academy>". I can set up email things (postfix, DKIM, DMARC, SPF) so that the email is signed as honestly coming from webwork-hosting.runestone.academy. Then the email reaches recipients and is not blocked as potential spam.

So I could take this approach to a PR for the distribution. But would it annoy people at "typical installations" to do this? Should it be configurable, so you can leave the "from" address as the sender's actual email address? Note that if we do that, and a person's email address in WeBWorK ends up as something outside of the institution's domain, they can end up with problems. So if we do that, do we also need to enforce that emails only come from some list of approved domains?

[dlglin]

I just ran into a related scenario: our institution used to whitelist internal servers so that mail sent from postfix would be delivered to institutional addresses, but they recently stopped allowing this, so I've been working on getting our WW server to use the University's smtp server.

The issue is that the smtp server requires authentication, and it will only send messages where the "from" address is the authenticated user. That means that I need something similar to what @Alex-Jordan describes, except instead of the "from" address being "Stu Student stu.student@myschool.edu" I need it to be "Stu Student generic.account@myschool.edu", where "generic.account" is the username of the account used to authenticate with the smtp server.

I'm assuming that there are still WW servers that are able to use postfix to send mail with the appropriate "from" address (particularly if they don't let students change their email address), so I think having the current behaviour as an option is probably still desired.

[Alex-Jordan]

I'm assuming that there are still WW servers that are able to use postfix to send mail with the appropriate "from" address (particularly if they don't let students change their email address), so I think having the current behaviour as an option is probably still desired.

What would be an "appropriate" from address in that situation?

[dlglin]

By "appropriate" I mean their actual email address.

[somiaj]

Could the Relpy-To be used in this case with the actual email address?

[dlglin]

That's what I did in my testing.

[Alex-Jordan]

By "appropriate" I mean their actual email address.

OK, but that's still a problem (I think). Say your actual email address is something@ucalgary.ca. And there is a WW server at webwork.ucalgary.ca using postfix. And that WW server sends an email "from" something@ucalgary.ca to the instructor whose email address is instructor@ucalgary.ca.

Then the mail server at ucalgary.ca receives the email, sees is is "from" ucalgary.ca, but also sees it actually came from somewhere else (webwork.ucalgary.ca) because it doesn't have the DKIM signature for ucalgary.ca. The mail server at ucalgary.ca can then declare it to be spam.

Of course if you get the mail server admins to whitelist webwork.ucalgary.ca, maybe it is OK. But today's mail server admin might be OK with that, and tomorrow's mail server admin doesn't trust WeBWorK/postfix as secure, and changes policy.

[Alex-Jordan]

If it's helpful to anyone, the patches I have for this are the following, with four new things in site.conf ($generic_sender_name, $feedback_sender_email, $instructor_sender_email, $notification_sender_email).

I think I haven't contributed it because it would take more work to make it configurable.

diff --git a/lib/Mojolicious/WeBWorK/Tasks/SendInstructorEmail.pm b/lib/Mojolicious/WeBWorK/Tasks/SendInstructorEmail.pm
index 44ba7a71f..eb4a7059c 100644
--- a/lib/Mojolicious/WeBWorK/Tasks/SendInstructorEmail.pm
+++ b/lib/Mojolicious/WeBWorK/Tasks/SendInstructorEmail.pm
@@ -117,8 +117,9 @@ sub mail_message_to_recipients ($job, $ce, $db, $mail_data) {
 }
 
 sub email_notification ($job, $ce, $mail_data, $result_message) {
+       my $from = $ce->{generic_sender_name} . ' <' . $ce->{notification_sender_email} . '>';
        my $email =
-               Email::Stuffer->to($mail_data->{defaultFrom})->from($mail_data->{defaultFrom})->subject('WeBWorK email sent')
+               Email::Stuffer->to($mail_data->{defaultFrom})->from($from)->subject('WeBWorK email sent')
                ->text_body($result_message)->header('X-Remote-Host' => $mail_data->{remote_host});
 
        eval {

diff --git a/lib/WeBWorK/ContentGenerator/Feedback.pm b/lib/WeBWorK/ContentGenerator/Feedback.pm
index dae2d403c..8fe42b283 100644
--- a/lib/WeBWorK/ContentGenerator/Feedback.pm
+++ b/lib/WeBWorK/ContentGenerator/Feedback.pm
@@ -182,10 +182,11 @@ $emailableURL
                if ($ce && $verbosity >= 2) {
                        $msg .= "***** Data about the environment: *****\n\n" . Dumper($ce) . "\n\n";
                }
-
+               my $from_name = $user ? $user->full_name : $ce->{generic_sender_name};
+               my $from_address = $ce->{feedback_sender_email};
                my $email =
-                       Email::Stuffer->to(join(',', @recipients))->from($sender)->subject($subject)->text_body($msg)
-                       ->header('X-Remote-Host' => $remote_host);
+                       Email::Stuffer->to(join(',', @recipients))->from("$from_name <$from_address>")->reply_to($sender)->subject($subject)
+                       ->text_body($msg)->header('X-Remote-Host' => $remote_host);
 
                # Extra headers
                $email->header('X-WeBWorK-Route',  $route)    if defined $route;

diff --git a/lib/WeBWorK/ContentGenerator/Instructor/SendMail.pm b/lib/WeBWorK/ContentGenerator/Instructor/SendMail.pm
index e4792cccd..7005d9330 100644
--- a/lib/WeBWorK/ContentGenerator/Instructor/SendMail.pm
+++ b/lib/WeBWorK/ContentGenerator/Instructor/SendMail.pm
@@ -216,6 +216,9 @@ sub initialize ($c) {
                                $c->param('savefilename', 'default.msg') if $c->param('savefilename');
                        }
                }
+               # Regardless of above, "$from" must use an address with an acceptable domain
+               $from = $db->getUser($user)->full_name . ' <' . $ce->{instructor_sender_email} . '>';
+
                $c->param('from',    $from)    if $from;
                $c->param('replyTo', $replyTo) if $replyTo;
                $c->param('subject', $subject) if $subject;

diff --git a/templates/ContentGenerator/Instructor/SendMail/main_form.html.ep b/templates/ContentGenerator/Instructor/SendMail/main_form.html.ep
index 719d408da..46d17d0f3 100644
--- a/templates/ContentGenerator/Instructor/SendMail/main_form.html.ep
+++ b/templates/ContentGenerator/Instructor/SendMail/main_form.html.ep
@@ -34,8 +34,8 @@
                                                <%= label_for from => maketext('From:'),
                                                        class => 'col-sm-3 col-form-label col-form-label-sm' =%>
                                                <div class="col-sm-9">
-                                                       <%= text_field from => $c->{from}, id => 'from',
-                                                               class => 'form-control form-control-sm' =%>
+                                                       <%= text_field from => $c->{from}, readonly => undef, id => 'from',
+                                                               class => 'form-control form-control-sm form-control-plaintext px-2' =%>
                                                </div>
                                        </div>
                                        <div class="row mb-1">

[dlglin]

Currently the Instructor Email form has a field for "Reply-To:" which gets pre-populated with the current user's name and email address, but this information isn't actually included in the message that is sent by the minion process.

In implementing what @Alex-Jordan included above I need to set the "Reply-to:" header in the message if $instructor_sender_email is defined. The question I have is what to do for the case where it's not set (i.e. the current default).

Options:

1. Maintain the current behaviour, which is to ignore whatever is typed in the "Reply-To:" field.
2. Always add the contents of the "Reply-To:" field as the reply-to address in the message, which will mean that in most cases the message will have the same "From:" and "Reply-To:" headers.
3. Change the form so that the "Reply-To:" field starts empty, and only set the "Reply-To:" header in the message if something is typed in that field.

[1eo1ui]

I am currently working on migrating WeBWorK's email functionality to integrate with our university's Azure-based email system using Microsoft Graph API. By registering an Azure app and utilizing the Graph API, we can handle email communication while ensuring compliance with security standards like SPF, DKIM, and DMARC, thus providing a more reliable and secure solution for sending emails.

One of the key advantages of using the Graph API is the ability to send emails with a Reply-To address. This allows emails to be sent from a verified domain (e.g., no-reply@university.edu), while replies are directed to the appropriate user, such as a student's or instructor’s personal or institutional email. This ensures that the email's origin remains trusted, mitigating potential issues with email deliverability or rejection by external mail services.

By moving WeBWorK's email handling to Azure through the Graph API, we can simplify mail management for institutions that do not have their own email servers or are using external systems like Postfix. This also helps improve email deliverability, particularly when sending to services like Gmail or Yahoo, which have strict requirements for email authenticity and validation.

Additionally, it's worth considering support for Microsoft Graph API in future versions of WeBWorK. As more institutions move to cloud-based systems like Azure, supporting Graph API will ensure seamless integration with modern infrastructures, reducing maintenance overhead for email services and providing a standardized, secure way to handle emails. This could enhance the flexibility and adaptability of WeBWorK in various deployment environments.