[BUG] Network delays when sending alert messages.

[dkubicahitachi]

Describe the bug
Network delays when sending alert messages.

Sometimes messages are not sent during tests of sending alert messages and an exception occurs. I suspect network delays/FW cause errors randomly. On the same machine, there is no problem with the same mailbox using Alertmanager.

[2023-07-06T22:12:19,684][INFO ][o.o.n.s.SendMessageActionHelper] [kubernetes-master-vm-0] notifications:getAllConfigs-get [wncJLYkBTr_U9fu7o619, v3QALYkBTr_U9fu7Sq-d] [2023-07-06T22:12:19,707][INFO ][o.o.n.c.c.DestinationSmtpClient] [kubernetes-master-vm-0] EmailException javax.mail.MessagingException: Exception reading response; nested exception is: java.net.SocketException: Connection reset [2023-07-06T22:12:19,707][INFO ][o.o.n.s.SendMessageActionHelper] [kubernetes-master-vm-0] notifications:sendMessage:statusCode=424, statusText=sendEmail Error, status:Exception reading response [2023-07-06T22:12:19,708][INFO ][o.o.n.s.SendMessageActionHelper] [kubernetes-master-vm-0] notifications:wncJLYkBTr_U9fu7o619:statusCode=424, statusText=sendEmail Error, status:Exception reading response [2023-07-06T22:12:19,708][WARN ][o.o.n.a.PluginBaseAction ] [kubernetes-master-vm-0] notifications:OpenSearchStatusException: org.opensearch.OpenSearchStatusException: {"event_status_list": [{"config_id":"iJOwKYgBPMWZN1s6JSE4","config_type":"email","config_name":"login_errors","email_recipient_status":[{"recipient":"user@mail.com","delivery_status":{"status_code":"424","status_text":"sendEmail Error, status:Exception reading response"}}],"delivery_status":{"status_code":"424","status_text":"sendEmail Error, status:Exception reading response"}}]} at org.opensearch.notifications.send.SendMessageActionHelper.executeRequest(SendMessageActionHelper.kt:99) ~[?:?] at org.opensearch.notifications.send.SendMessageActionHelper$executeRequest$1.invokeSuspend(SendMessageActionHelper.kt) ~[?:?] at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33) [kotlin-stdlib-1.6.10.jar:1.6.10-release-923(1.6.10)] at kotlinx.coroutines.internal.ScopeCoroutine.afterResume(Scopes.kt:32) [kotlinx-coroutines-core-jvm-1.4.3.jar:?] at kotlinx.coroutines.AbstractCoroutine.resumeWith(AbstractCoroutine.kt:113) [kotlinx-coroutines-core-jvm-1.4.3.jar:?] at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:46) [kotlin-stdlib-1.6.10.jar:1.6.10-release-923(1.6.10)] at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:106) [kotlinx-coroutines-core-jvm-1.4.3.jar:?] at kotlinx.coroutines.scheduling.CoroutineScheduler.runSafely(CoroutineScheduler.kt:571) [kotlinx-coroutines-core-jvm-1.4.3.jar:?] at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.executeTask(CoroutineScheduler.kt:750) [kotlinx-coroutines-core-jvm-1.4.3.jar:?] at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.runWorker(CoroutineScheduler.kt:678) [kotlinx-coroutines-core-jvm-1.4.3.jar:?] at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.run(CoroutineScheduler.kt:665) [kotlinx-coroutines-core-jvm-1.4.3.jar:?] [2023-07-06T22:12:19,709][ERROR][o.o.n.a.SendTestNotificationAction] [kubernetes-master-vm-0] notifications:SendTestNotificationAction-send Error:OpenSearchStatusException[{"event_status_list": [{"config_id":"iJOwKYgBPMWZN1s6JSE4","config_type":"email","config_name":"login_errors","email_recipient_status":[{"recipient":"user@mail.com","delivery_status":{"status_code":"424","status_text":"sendEmail Error, status:Exception reading response"}}],"delivery_status":{"status_code":"424","status_text":"sendEmail Error, status:Exception reading response"}}]}]

To Reproduce
Steps to reproduce the behavior:

1. Turn on firewall
2. Create a channel with SMTP sender
3. Send test message multiple times

Expected behavior
Messages are always sent.

Environment (please complete the following information):

• OS: [Ubuntu 20.04.5 LTS]
• Version [v 2.8.0]