-
Notifications
You must be signed in to change notification settings - Fork 265
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
HiveMQ CE Subscribers Become Idle Under Heavy Load with Shared Subscriptions and QoS=AtLeastOnce #558
Comments
Thank you for bringing this to our attention and for providing such comprehensive materials alongside the issue - it’s greatly appreciated. We will attempt to reproduce this issue internally. If successful, we’ll prioritize and schedule a bug fix. Should we encounter any difficulties reproducing the issue, we may reach out to you for additional clarification or details. Please note that high-load and demanding use cases are best suited for the HiveMQ Enterprise Edition, which offers advanced features designed to ensure enterprise-grade reliability, even under the most challenging circumstances. Thank you again, and have a great day! Best regards, |
I am working with Vladimir and also currently our client that has this deployment and where identified the issue also brought it up with HiveMQ representatives and they agreed to help to support with this issue. Please feel free to ping us and we can set up a call we can show the issue there live. |
Hi @VladimirMakarevich & @pesetskyps, I had reproducer active for over an hour at a time and for multiple shorter sessions, but was unable to observe the behaviour you captured in the video you shared. It would be interesting to see what the broker reports from its side. Can I ask you to inspect event.log for mentions of your subscribing client? In my own tests, I see dropped messages, as the 20 to 1 ratio is overwhelming the single subscriber, but generally message flow never comes to a halt. As we have observed similar behaviour in the past with some libraries, can I ask you to replace the subscribing client with an instance of HiveMQ's MQTT CLI? Kind regards, |
Sure @FinnHMQ we will do that. For event logs you can check the logs that Vladimir attached URL to, it has all possible logs that we would capture from HiveMQ including event.log If you can't reproduce the behaviour on your side I recommend us having a call, because we can show you the issue live. But let us check what you've asked first |
Hi @FinnHMQ. Thank you for your suggestions. I tested using HiveMQ's MQTT CLI as you recommended. During my attempts to reproduce the issue, I periodically encountered the following error:
After which the consumer switched to idle behavior. But most likely consumer just disconnects from the broker in this case. I have created a screencast demonstrating this error and collected all relevant broker logs. You can access them here: hivemq-ce-idle-issue/diagnostics and logs/v4. I'm unsure whether this error is related to the issues we're experiencing with the python aiomqtt library, the js mqtt library, or the .net mqttnet library, or if it indicates a new issue. Please let me know if you need any additional information. Best regards, |
while I still wasn't able to reproduce the behaviour directly with your shared code, the data you shared (especially the run with the MQTT CLI) gave us valuable insights. The disconnect by the consumer hints at a protocol error occurring and the broker erroneously duplicating a message. We have made progress with our test suite regarding the behaviour close to end of day yesterday. I will keep you updated with our findings. Thank you for bringing this to our attention. Kind regards, |
Hi @FinnHMQ is there any news on the testing and potential fix for the issue that you was able to reproduce? Thank you for your support |
Could you please test the HiveMQ MQTT CLI command below in your environment to replicate the steps of your .NET Subscriber application and check if the same behavior occurs?
Best regards |
Thank you for providing the MQTT CLI command. I ran the command in my environment to replicate the steps. Unfortunately, I am still encountering the following error:
To assist in diagnosing the issue, I have added logs from the latest test runs, which you can review here. Additionally, I have included a video demonstration and a screenshot showing the exact command execution for your reference. Please let me know if there are any additional steps I should take or if you need further information to help resolve this issue. Best regards, |
Please avoid using MQTT CLI Shell Mode and instead run the command directly in the terminal.
Best regards, |
Thank you for clarifications. Unfortunately this didn't help, but at least you can see more logs. I have added new logs v6 from the latest test runs, you can review them here. And as usual, I have included a video demonstration. Please let me know if there are any additional steps I should take. PS: I would also like to note that I have also tested this command on HiveMQ Enterprise and the free Mosquitto broker and this error does not occur. Best regards, |
Expected behavior
Subscribers should consistently receive messages as long as the connection is active and the broker has messages on the subscribed topics, regardless of load conditions.
Actual behavior
After an initial period of normal message flow, subscribers become idle and do not receive any further messages until they are restarted.
To Reproduce
Steps
Reproducer code
How to run the .NET Publishers and Consumer
Details
What We Have Observed & Collected:
Publishers:
cleanStart = false
,protocolVersion = 5
Subscribers:
cleanStart = false
,protocolVersion = 5
The text was updated successfully, but these errors were encountered: