Gateway Bridge does randomly not reconnect

[mullerch]

• The issue is present in the latest release.
• I have searched the issues of this repository and believe that this is not a duplicate.

What happened?

Upon MQTT broker restart, the Gateway Bridge does sometime not reconnect, event when disconnection is detected. However, it still tries to publish event and acts as everything was fine.

What did you expect?

The Gateway Bridge tries indefinitely to reconnect when it's not connected.

Steps to reproduce this issue

Steps:

1. Run a MQTT server
2. Run the gateway bridge
3. Restart the MQTT server
4. Check the gateway bridge logs

Note: the issue does not occur at each run

Could you share your log output?

• usergw: user used to connect teh ChripStack Gateway Bridge
• user: user used to connect a generic MQTT client (MQTT explorer)

1. QOS 0, no reconnect

Broker log

1643881421: mosquitto version 2.0.14 starting
1643881421: Config loaded from /mosquitto/config/mosquitto.conf.
1643881421: Opening ipv4 listen socket on port 1883.
1643881421: Opening ipv6 listen socket on port 1883.
1643881421: mosquitto version 2.0.14 running
1643881452: New connection from 10.10.236.191:60420 on port 1883.
1643881452: New client connected from 10.10.236.191:60420 as mqtt-explorer-3b35f5df (p2, c1, k60, u'user').
1643881610: New connection from 10.10.236.191:32794 on port 1883.
1643881612: New client connected from 10.10.236.191:32794 as auto-18F89536-7318-88C0-E8CC-A3C179A59AEC (p2, c1, k30, u'usergw').
### BROKER RESTART REQUEST
1643881631: mosquitto version 2.0.14 terminating
1643881633: mosquitto version 2.0.14 starting
1643881633: Config loaded from /mosquitto/config/mosquitto.conf.
1643881633: Opening ipv4 listen socket on port 1883.
1643881633: Opening ipv6 listen socket on port 1883.
1643881633: mosquitto version 2.0.14 running
1643881663: New connection from 10.10.236.191:59894 on port 1883.
1643881663: New client connected from 10.10.236.191:59894 as mqtt-explorer-3b35f5df (p2, c1, k60, u'user').

Gateway Bridge log

"2022-02-03T09:46:50.402942991Z" level=info msg="starting ChirpStack Gateway Bridge" docs="https://www.chirpstack.io/gateway-bridge/" version=3.13.2
"2022-02-03T09:46:50.431104297Z" level=info msg="backend/semtechudp: starting gateway udp listener" addr="127.0.0.1:1700"
"2022-02-03T09:46:52.49205112Z" level=warning msg="[store]    memorystore wiped" module=mqtt
"2022-02-03T09:46:52.508691666Z" level=info msg="integration/mqtt: connected to mqtt broker"
"2022-02-03T09:46:58.576787018Z" level=info msg="integration/mqtt: subscribing to topic" qos=0 topic="gateway/fcc23dfffeaabbcc/command/#"
"2022-02-03T09:46:58.665580408Z" level=info msg="integration/mqtt: publishing state" gateway_id=fcc23dfffeaabbcc qos=0 state=conn topic=gateway/fcc23dfffeaabbcc/state/conn
### BROKER RESTART REQUEST
"2022-02-03T09:47:11.755604247Z" level=error msg="[client]   Connect comms goroutine - error triggered EOF" module=mqtt
"2022-02-03T09:47:11.759228609Z" level=error msg="mqtt: connection error" error=EOF
"2022-02-03T09:47:28.559118567Z" level=info msg="integration/mqtt: publishing event" event=stats qos=0 stats_id=c96542b6-f7e9-4ce7-868e-1ec56dfa56da topic=gateway/fcc23dfffeaabbcc/event/stats
"2022-02-03T09:47:58.548870359Z" level=info msg="integration/mqtt: publishing event" event=stats qos=0 stats_id=d864138d-9a02-428e-8274-e9b572330508 topic=gateway/fcc23dfffeaabbcc/event/stats
### BRIDGE STOP REQUEST
"2022-02-03T09:48:18.921323279Z" level=info msg="signal received" signal=terminated
"2022-02-03T09:48:18.921723835Z" level=warning msg="shutting down server"
"2022-02-03T09:48:18.922332548Z" level=info msg="integration/mqtt: publishing state" gateway_id=fcc23dfffeaabbcc qos=0 state=conn topic=gateway/fcc23dfffeaabbcc/state/conn
"2022-02-03T09:48:18.922995341Z" level=warning msg="[client]   Disconnect() called but not connected (disconnected/reconnecting)" module=mqtt

Gateway Bridge Configuration

[integration.mqtt.auth.generic]

  # MQTT servers.
  #
  # Configure one or multiple MQTT server to connect to. Each item must be in
  # the following format: scheme://host:port where scheme is tcp, ssl or ws.
  server="ssl://broker-url:1883"

  # Connect with the given username (optional)
  username="usergw"

  # Connect with the given password (optional)
  password="password"
  

2. QOS 1, no reconnect on longer duration

Broker log

1643882847: mosquitto version 2.0.14 starting
1643882847: Config loaded from /mosquitto/config/mosquitto.conf.
1643882847: Opening ipv4 listen socket on port 1883.
1643882847: Opening ipv6 listen socket on port 1883.
1643882847: mosquitto version 2.0.14 running
1643882871: New connection from 10.10.236.191:34416 on port 1883.
1643882871: Client mqtt-explorer-3b35f5df disconnected, not authorised.

Gateway Bridge log

"2022-02-03T10:07:25.77090741Z" level=error msg="[client]   Connect comms goroutine - error triggered EOF" module=mqtt
"2022-02-03T10:07:25.776744786Z" level=error msg="mqtt: connection error" error=EOF
"2022-02-03T10:07:25.883611536Z" level=error msg="[client]   EOF" module=mqtt
"2022-02-03T10:07:25.909225095Z" level=warning msg="[client]   failed to connect to broker, trying next" module=mqtt
"2022-02-03T10:07:36.114944556Z" level=info msg="integration/mqtt: publishing event" event=stats qos=1 stats_id=76ed29b6-de9f-4a21-9561-f6f5459bdba5 topic=gateway/fcc23dfffeaabbcc/event/stats
"2022-02-03T10:08:06.12088347Z" level=info msg="integration/mqtt: publishing event" event=stats qos=1 stats_id=8c6b5bcb-ab1a-4bf1-9f45-df150b219fc8 topic=gateway/fcc23dfffeaabbcc/event/stats
"2022-02-03T10:08:36.123365222Z" level=info msg="integration/mqtt: publishing event" event=stats qos=1 stats_id=d200c86b-c741-458f-8234-d0ac061cb9d6 topic=gateway/fcc23dfffeaabbcc/event/stats
"2022-02-03T10:09:06.142791908Z" level=info msg="integration/mqtt: publishing event" event=stats qos=1 stats_id=92096d76-c0b9-4814-8bb0-7f1f426e844a topic=gateway/fcc23dfffeaabbcc/event/stats
"2022-02-03T10:09:36.135940902Z" level=info msg="integration/mqtt: publishing event" event=stats qos=1 stats_id=d5543524-2365-4b2b-8860-4e83cf039cc1 topic=gateway/fcc23dfffeaabbcc/event/stats

Gateway Bridge Configuration

      [integration.mqtt.auth.generic]

        # MQTT servers.
        #
        # Configure one or multiple MQTT server to connect to. Each item must be in
        # the following format: scheme://host:port where scheme is tcp, ssl or ws.
        server="ssl://broker-url:1883"

        # Connect with the given username (optional)
        username="usergw"

        # Connect with the given password (optional)
        password="password"

        qos=1

Workaround

Use terminate_on_connect_error=true and supervise the gateway bridge with 'monit'.

Your Environment

• ChirpStack Gateway Bridge: v3.13.2
• Mosquitto: v2.0.14

[jxc876]

Might be related to #206