[2.1 & 3.0]: Email 7-bit format is not checking for 1000 byte limits

[sbulen]

Basic Information

In the sendmail/mimespecialchars routines, there is no check for a character limit.

I believe the 7-bit format has a 1000 byte limit that should be honored.
https://stackoverflow.com/questions/25710599/content-transfer-encoding-7bit-or-8-bit

I suspect this may be causing the "Message has lines too long for transport" error reported in the forum.
https://www.simplemachines.org/community/index.php?topic=589251.0

Steps to reproduce

Tricky to reproduce... My host appears to simply strip the 7-bit portion from the email & use the utf8 portion.

User reports getting it when flagging a post as a forum announcement, to trigger an email. Some emails are delivered, some get undeliverable replies with the "Message has lines too long for transport" message.

What is easy to reproduce is to see that SMF's sendmail produces lines longer than 1000 bytes...

E.g., this 198-character line in Ukranian:
Восени 2020-го року, я вирішив, що технічно формат проекту не відповідає вимогам часу. Тому рішуче почав перебудовувати сайт на новий движок - WordPress. Планувалось, та й так і сталося, що найбільш

Becomes this 1080 byte line in the 7bit chunk of an SMF email body:
Восени 2020-го року, я вирішив, що технічно формат проекту не відповідає вимогам часу. Тому рішуче почав перебудовувати сайт на новий движок - WordPress. Планувалось, та й так і сталося, що найбільш

As a result, the 1000 byte limitation is much tougher on languages with multibyte characters.

Expected result

No response

Actual result

No response

Version/Git revision

2.1.4

Database Engine

MySQL

Database Version

8.4

PHP Version

8.3.8

Logs

Additional Information

Lots of details in this thread:
https://www.simplemachines.org/community/index.php?topic=589251.0

[sbulen]

I wonder if we should abandon the 7-bit representation altogether, and just keep the utf8 one.

Otherwise we must insert linebreaks into the source text, which would be ugly...

[jdarwood007]

I suspect UTF-8 is about 100% easier to do now, and we would have a near 0% chance of breaking anyone's email since every email client in the last 20 years supports UTF-8.

[sbulen]

The good news is that our utf8 works already.

The difficult question is how safe it is to remove the old 7-bit...

You know SOMEBODY out there has users running an ancient AOL client in windows xp...

[jdarwood007]

I would say for 3.0, remove 7bit, for 2.1 we leave it, breaking the standard.

[Sesquipedalian]

Anyone running an email client that still can't support UTF-8 in 2024 would already be seeing lots of unreadable email messages. After all, if your own email service provider is simply deleting the 7-bit part in transit, @sbulen, it probably isn't the only one. I think we can kill it at our earliest convenience.

[sbulen]

Related discussion in the forum... The 7-bit limitation also affects subject lines, which, for email purposes, are restricted to 100 bytes, which can be ~50 characters in some languages:
https://www.simplemachines.org/community/index.php?topic=589350.0

[jdarwood007]

@sbulen I don't know enough about mimespecialchars to be able to send a PR here, but I want to get one going for 3.0

I suspect what I need to do is change 7bit to 8bit, but I can't see what else would need to be adapted in the code. The Mail::mimespecialchars function passes the string into Utils::entityDecode, but that just decodes the HTML entities. There is a function for handling the UTF-8 stuff by base64 encoding it, but that would return base64 for the encoding.

I also think that while doing this PR, it's a great time to finally do it... Remove the legacy Hotmail fix. If the 3 email carriers we have listed still don't have this fixed, that is their problem.

[sbulen]

I don't think we need to change anything, actually...

The 7-bit is added to all emails as a fallback. We need to remove the weird 7-bit stuff which is a 2nd copy of the message, & leave the rest alone.

I tried to explain an approach here...
https://www.simplemachines.org/community/index.php?msg=4189946

[jdarwood007]

Ahh, I thought you had tested something. So it looks like all you are suggesting is we dump the second copy, which is the non-UTF-8 copy. You dumped the boundary, but I think that needs to be kept for the plaintext vs HTML copy.

Although it seems he had an issue with sendmail. Which makes me wonder if we need to keep the 7bit or define it for 8bit to have sendmail process it.

[sbulen]

No, I think the 7-bit needs to go.

Otherwise you get the LOOOONG lines as shown above, which break the 1000-byte rule. The encoded blocks don't have that problem, they never exceed 80 bytes.

The limitation is to the email raw format, not the end format.

The "Message has lines too long for transport" error has been reported elsewhere over the years also, it's not just that one guy. That guy was technical enough to help define the issue though.

One simple encoded block, IMO.

l also think that (in ancient times) SMF limited subject lengths specifically so the whole subject could appear in email. Subject lines need a review as well. And - more importantly - corresponding limitations removed from topic names.

It can be VERY restrictive for folks with multi-byte chars. (link above https://www.simplemachines.org/community/index.php?topic=589350.0)