Nodes fail every other connect (i.e.: deploy)

[pdreker]

I am running these nodes against a 3 Node RabbitMQ cluster on a Raspberry Pi Kubernetes Cluster.

When re-deploying there seems to be some kind of race condition: If the flow is currently running, then changed and re-deployed I receive the following error in the Node Red debug console:

"Could not consume message: Error: Channel ended, no reply will be forthcoming"

When the AMQP in node is set to "Exclusive" there is a different message:

"Could not consume message: Error: Operation failed: QueueDeclare; 405 (RESOURCE-LOCKED) with message "RESOURCE_LOCKED - cannot obtain exclusive access to locked queue 'prometheus' in vhost 'homeautomation'. It could be originally declared on another connection or the exclusive property value does not match that of the original declaration.""

In the latter case there are no other consumers on the queue.

In both cases it seems like the disconnect/reconnect is happening so quick, that the broker has not completely cleaned up in between. The Node then either shows disconnected (waited for a few minutes - no change) or actually in case of the first error message shows "connected", but does not receive any messages (exactly as the error message implies: "No messages will be forthcoming"). In both cases the flow does not work as a result and does so in a rather subtle way.

Simply changing the flow (moving some node is enough) and then redeploying makes the connection work.

Maybe there is a retry missing for certain conditions, or maybe even a small (configurable?) delay before actually connecting could solve this... I think...