[Questions] There are quorum queues that have lost a node online

[kamilbro5]

Community Support Policy

• I have read RabbitMQ's Community Support Policy
• I run RabbitMQ 4.x, the only series currently covered by community support
• I promise to provide all relevant information (versions, logs from all nodes, rabbitmq-diagnostics output, detailed reproduction steps)

RabbitMQ version used

4.1.1

Erlang version used

27.3.x

Operating system (distribution) used

Red Hat Enterprise Linux 9.6

How is RabbitMQ deployed?

RPM package

rabbitmq-diagnostics status output

See https://www.rabbitmq.com/docs/cli to learn how to use rabbitmq-diagnostics

N/A

Logs from node 1 (with sensitive values edited out)

See https://www.rabbitmq.com/docs/logging to learn how to collect logs

There are no logs when the event occurs

Logs from node 2 (if applicable, with sensitive values edited out)

See https://www.rabbitmq.com/docs/logging to learn how to collect logs

There are no logs when the event occurs

Logs from node 3 (if applicable, with sensitive values edited out)

See https://www.rabbitmq.com/docs/logging to learn how to collect logs

There are no logs when the event occurs

rabbitmq.conf

See https://www.rabbitmq.com/docs/configure#config-location to learn how to find rabbitmq.conf file location

default_vhost = ZT002_api_rmq


# Log Settings
log.dir = /app/logs/rmq/node1
log.file = mylocalhostname
log.file.level = debug
log.file.rotation.date = $D0
log.file.rotation.count = 120
listeners.tcp.default = 4443
log.connection.level = info
log.channel.level = info
log.queue.level = info
log.mirroring.level = info
log.federation.level = info
log.upgrade.level = info
log.default.level = info

# SSL Settings
listeners.ssl.default = 9443
ssl_options.certfile = /etc/szef/requested_certs/mylocalhostname.crt
ssl_options.keyfile = /etc/szef/requested_certs/mylocalhostname.key
ssl_options.fail_if_no_peer_cert = false
ssl_options.verify = verify_none
ssl_options.versions.1 = tlsv1.2
ssl_options.honor_cipher_order = true
ssl_options.honor_ecc_order = true
ssl_options.ciphers.1 = DHE-RSA-AES128-GCM-SHA256
ssl_options.ciphers.2 = DHE-RSA-AES128-SHA256
ssl_options.ciphers.3 = DHE-RSA-AES256-GCM-SHA384
ssl_options.ciphers.4 = DHE-RSA-AES256-SHA256
ssl_options.ciphers.5 = ECDHE-RSA-AES128-GCM-SHA256
ssl_options.ciphers.6 = ECDHE-RSA-AES128-SHA256
ssl_options.ciphers.7 = ECDHE-RSA-AES256-GCM-SHA384
ssl_options.ciphers.8 = ECDHE-RSA-AES256-SHA384

# MGMT Settings
#management.tcp.port = 9060
management.ssl.port = 8443
management.ssl.idle_timeout =  120000
management.ssl.inactivity_timeout =  120000
management.ssl.request_timeout = 10000
management.ssl.certfile = /etc/szef/requested_certs/mylocalhostname.crt
management.ssl.keyfile = /etc/szef/requested_certs/mylocalhostname.key
management.ssl.versions.1 = tlsv1.2
management.ssl.honor_cipher_order = true
management.ssl.honor_ecc_order = true
management.ssl.client_renegotiation = false
management.ssl.secure_renegotiate = true
management.csp.policy = default-src 'self' *.mylocalhostname 'unsafe-eval' 'unsafe-inline' frame-ancestors 'none'
management.hsts.policy = max-age=31536000; includeSubDomains
management.cors.allow_origins.1 = *.mylocalhostname
management.cors.allow_origins.2 = *.mylocalhostname
management.cors.max_age = 3600
management.headers.content_type_options = nosniff
management.headers.xss_protection = 1; mode=block
management.headers.frame_options = SAMEORIGIN
management.ssl.ciphers.1 = DHE-RSA-AES128-GCM-SHA256
management.ssl.ciphers.2 = DHE-RSA-AES128-SHA256
management.ssl.ciphers.3 = DHE-RSA-AES256-GCM-SHA384
management.ssl.ciphers.4 = DHE-RSA-AES256-SHA256
management.ssl.ciphers.5 = ECDHE-RSA-AES128-GCM-SHA256
management.ssl.ciphers.6 = ECDHE-RSA-AES128-SHA256
management.ssl.ciphers.7 = ECDHE-RSA-AES256-GCM-SHA384
management.ssl.ciphers.8 = ECDHE-RSA-AES256-SHA384
management.sample_retention_policies.global.minute  = 5
management.sample_retention_policies.global.hour    = 60
management.sample_retention_policies.global.day     = 1200
management.sample_retention_policies.basic.minute 	= 5
management.sample_retention_policies.basic.hour   	= 60
management.sample_retention_policies.detailed.10 	= 5

default_user_tags.administrator = true
default_user_tags.management = true

# DISK limits
disk_free_limit.relative = 0.80

# MEM Settings
vm_memory_calculation_strategy = rss
vm_memory_high_watermark.relative = 0.87
vm_memory_high_watermark_paging_ratio = 0.75

# LDAP Settings
auth_backends.1 = internal
auth_backends.2 = cache
auth_cache.cached_backend = ldap
auth_cache.cache_ttl = 300000
auth_ldap.servers.1  = ldap2012.mylocalhostname
auth_ldap.user_dn_pattern = CN=${username},CN=A,DC=A,DC=A
auth_ldap.use_ssl    = true
auth_ldap.port       = 636
auth_ldap.log        = false
auth_ldap.timeout    = 30000
auth_ldap.idle_timeout = 18000


# Quorum queue properties
quorum_queue.continuous_membership_reconciliation.enabled = true
quorum_queue.continuous_membership_reconciliation.target_group_size = 3
quorum_queue.continuous_membership_reconciliation.auto_remove = true

# Custom config from databags
disk_free_limit.relative = 0.80
cluster_partition_handling  = autoheal
message_interceptors.incoming.set_header_timestamp.overwrite = false

Steps to deploy RabbitMQ cluster

yum install rabbitmq

Steps to reproduce the behavior in question

My cluster behaves correctly after startup.
After connecting clients, online nodes drop out of the queues at random times.
A dotNET/.NET client is connected to the queues.
When an online node drops out of the queue (it's always a random node), there are no entries in the main log, connection log, or queue.log

I see a small peak of messages when a node drops out of the queue, about 30 messages at a time, I monitor it

advanced.config

See https://www.rabbitmq.com/docs/configure#config-location to learn how to find advanced.config file location

[
  {rabbit, [
      {config_entry_decoder, [
             {passphrase,{ file, "/app/rmq/node1/config_pass"} },
             {cipher, blowfish_cfb64},
             {hash, sha256},
             {iterations, 10000}
         ]},
      {credential_validator, [{validation_backend, rabbit_credential_validator_password_regexp},
                                   {regexp, <<"^[a-zA-Z0-9\S$]{2,100}">>}]
        },
      {default_user, admin},
      {default_pass,
             {encrypted, <<"AAAAAAAAAAAAAAAAAAAAAA">> }}
    ]},
   {rabbitmq_auth_backend_ldap,[
     {group_lookup_base, "DC=A,DC=A"},
     {ssl_options, [
                    {cacertfile,"/etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem"},
                    {verify, verify_peer},
                    {fail_if_no_peer_cert, false}

     ]},
     {vhost_access_query, {'or', [{in_group, "CN=A,OU=Distribution,OU=Groups,OU=_Exchange,DC=A,DC=A"},
                          {in_group, "CN=!${vhost}_Users,OU=RabbitMQ,OU=Groups,OU=A,DC=A,DC=A"},
                          {in_group, "CN=!${vhost}_Users,OU=RabbitMQ,OU=Groups,OU=A,DC=A,DC=A"}]} },
     {tag_queries, [{administrator, {in_group, "CN=A,OU=Distribution,OU=Groups,OU=_Exchange,DC=A,DC=A"}},
                    {monitoring, {in_group, "CN=!${vhost}_Users,OU=RabbitMQ,OU=Groups,OU=A,DC=A,DC=A"}},
                    {management, {in_group, "CN=!${vhost}_Users,OU=RabbitMQ,OU=Groups,OU=A,DC=A,DC=A"}}
                    ] }    ]},
    {lager, [
        {error_logger_hwm, 1024}
    ]}
].

Application code

# PASTE CODE HERE, BETWEEN BACKTICKS

Kubernetes deployment file

# Relevant parts of K8S deployment that demonstrate how RabbitMQ is deployed
# PASTE YAML HERE, BETWEEN BACKTICKS

What problem are you trying to solve?

rabbitmq-queues delete_member -p / Crm_TransactionMessage node2@hostname
rabbitmq-queues add_member -p / Crm_TransactionMessage node2@hostname
or
restart nodes...
after 1-3 days production operation still node leave queueus

[michaelklishin]

We cannot guess what happens to your node. We are engineers, not magicians.

See node logs on every node for clues.