IoT Hub Disconnect

[LinderPi]

Development Machine, OS, Compiler (and Other Relevant Toolchain Info)

Embedded Linux on i.MX 8M Mini
Cross Compiled with Yocto Kirkstone on Ubuntu 22.04

SDK Version (Please Give Commit SHA if Manually Compiling)

LTS_07_2022, but also with LTS_08_2024

Protocol

MQTT

Describe the Bug

I have a device that provisions successfully to the IoT hub and is able to send messages. After a few minutes, the problem appears and I get message send timeouts. This is shown in the send callback (registered with IoTHubDeviceClient_LL_SendEventAsync) with the error IOTHUB_CLIENT_CONFIRMATION_MESSAGE_TIMEOUT. At the same time, the Azure SDK prints an error about disconnecting the MQTT connection. Also, in the connection status callback (registered with IoTHubDeviceClient_LL_SetConnectionStatusCallback) the error IOTHUB_CLIENT_CONNECTION_COMMUNICATION_ERROR appears. This happens multiple times (10-15) before another error from the SDK states a failure opening the connection to the endpoint. Another error shows that even this disconnection does not work. After 30 seconds, the connection can again be established. But the same procedure happens again and again.

In the IoT hub are some errors 400_027 meaning that a connection is forcefully closed on a new connection: https://learn.microsoft.com/en-us/azure/iot-hub/troubleshoot-error-codes#400027-connection-forcefully-closed-on-new-connection. However, there is no new connection established with the same device identity. I suppose the problem to be somewhere else, as it worked for more than a year.

Console Logs

06:58:48.578533 - info: Telemetry: Message sent failed, buffer message
06:58:48.580350 - warning: provider/Provider: Timeout without matching id received
06:58:48.580958 - warning: provider/Provider: Timeout for id {xxx}
...
06:59:48.303136 - warning: provider/Worker: Message sent failed 2
06:59:48.303172 - warning: provider/Provider: Timeout without matching id received
Error: Time:Wed Aug 28 06:59:48 2024 File:/usr/src/debug/azure-iot-sdk/lts_07_2022+gitAUTOINC+3fd808bee3-r0/git/iothub_client/src/iothubtransport_mqtt_common.c Func:ProcessPendingTelemetryMessages Line:2483 Disconnecting MQTT connection because message PUBACK (4) timeout.
06:59:50.808574 - debug: provider/Worker: IoTHub disconnected 5
...
Error: Time:Wed Aug 28 06:59:50 2024 File:/usr/src/debug/azure-iot-sdk/lts_07_2022+gitAUTOINC+3fd808bee3-r0/git/umqtt/src/mqtt_client.c Func:onOpenComplete Line:452 Error: failure opening connection to endpoint
...
Error: Time:Wed Aug 28 07:00:21 2024 File:/usr/src/debug/azure-iot-sdk/lts_07_2022+gitAUTOINC+3fd808bee3-r0/git/iothub_client/src/iothubtransport_mqtt_common.c Func:UpdateMqttConnectionStateIfNeeded Line:2844 mqtt_client timed out waiting for CONNACK: disconnecting MQTT connection
07:00:22.047308 - debug: provider/Worker: IoTHub connected
07:00:22.047604 - info: provider/Provider: Connection status changed, connected: true