[Questions] Api is very slow after stop one node from 3-nodes cluster

[kamilbro5]

Community Support Policy

• I have read RabbitMQ's Community Support Policy
• I agree to provide all relevant information (versions, logs, rabbitmq-diagnostics output, detailed reproduction steps)

RabbitMQ version used

3.13.7

Erlang version used

26.1.x

Operating system (distribution) used

RHEL9

How is RabbitMQ deployed?

RPM package

rabbitmq-diagnostics status output

See https://www.rabbitmq.com/docs/cli to learn how to use rabbitmq-diagnostics

# PASTE OUTPUT HERE, BETWEEN BACKTICKS

Logs from node 1 (with sensitive values edited out)

See https://www.rabbitmq.com/docs/logging to learn how to collect logs

# PASTE LOG HERE, BETWEEN BACKTICKS

Logs from node 2 (if applicable, with sensitive values edited out)

See https://www.rabbitmq.com/docs/logging to learn how to collect logs

# PASTE LOG HERE, BETWEEN BACKTICKS

Logs from node 3 (if applicable, with sensitive values edited out)

See https://www.rabbitmq.com/docs/logging to learn how to collect logs

# PASTE LOG HERE, BETWEEN BACKTICKS

rabbitmq.conf

See https://www.rabbitmq.com/docs/configure#config-location to learn how to find rabbitmq.conf file location

default_vhost = cluster


listeners.tcp.default = 4443
# SSL Settings
listeners.ssl.default = 9443
ssl_options.certfile = /app/rmq/ssl/name.pem
ssl_options.keyfile = /app/rmq/ssl/name.key
ssl_options.fail_if_no_peer_cert = false
ssl_options.verify = verify_none
ssl_options.versions.1 = tlsv1.2
ssl_options.honor_cipher_order = true
ssl_options.honor_ecc_order = true
ssl_options.ciphers.1 = DHE-RSA-AES256-GCM-SHA384
ssl_options.ciphers.2 = DHE-RSA-AES256-GCM-SHA384
ssl_options.ciphers.3 = DHE-RSA-AES128-GCM-SHA256
ssl_options.ciphers.4 = ECDHE-RSA-AES256-GCM-SHA384
ssl_options.ciphers.5 = ECDHE-RSA-AES128-GCM-SHA256
ssl_options.ciphers.6 = DHE-RSA-AES256-GCM-SHA384
ssl_options.ciphers.7 = DHE-RSA-AES128-GCM-SHA256
ssl_options.ciphers.8 = ECDHE-RSA-AES256-GCM-SHA384
ssl_options.ciphers.9 = ECDHE-RSA-AES128-GCM-SHA256
ssl_options.ciphers.10 = DHE-RSA-AES256-SHA256
ssl_options.ciphers.11 = DHE-RSA-AES128-SHA256
ssl_options.ciphers.12 = ECDHE-RSA-AES256-SHA384
ssl_options.ciphers.13 = ECDHE-RSA-AES128-SHA256

# MGMT Settings
#management.tcp.port = 9060
management.ssl.port = 8443
management.ssl.certfile = /app/rmq/ssl/name.pem
management.ssl.keyfile = /app/rmq/ssl/name.key
management.ssl.versions.1 = tlsv1.2
management.csp.policy = default-src 'self' *.centrala.test 'unsafe-eval' 'unsafe-inline'
management.hsts.policy = max-age=31536000; includeSubDomains
management.cors.allow_origins.1 = *.centrala.test
management.cors.allow_origins.2 = *.t.it.test
management.headers.content_type_options = nosniff
management.headers.xss_protection = 1; mode=block
management.headers.frame_options = SAMEORIGIN
management.ssl.ciphers.1 = DHE-RSA-AES256-GCM-SHA384
management.ssl.ciphers.2 = DHE-RSA-AES256-GCM-SHA384
management.ssl.ciphers.3 = DHE-RSA-AES128-GCM-SHA256
management.ssl.ciphers.4 = ECDHE-RSA-AES256-GCM-SHA384
management.ssl.ciphers.5 = ECDHE-RSA-AES128-GCM-SHA256
management.ssl.ciphers.6 = DHE-RSA-AES256-GCM-SHA384
management.ssl.ciphers.7 = DHE-RSA-AES128-GCM-SHA256
management.ssl.ciphers.8 = ECDHE-RSA-AES256-GCM-SHA384
management.ssl.ciphers.9 = ECDHE-RSA-AES128-GCM-SHA256
management.ssl.ciphers.10 = DHE-RSA-AES256-SHA256
management.ssl.ciphers.11 = DHE-RSA-AES128-SHA256
management.ssl.ciphers.12 = ECDHE-RSA-AES256-SHA384
management.ssl.ciphers.13 = ECDHE-RSA-AES128-SHA256
management.sample_retention_policies.global.minute  = 5
management.sample_retention_policies.global.hour    = 60
management.sample_retention_policies.global.day     = 1200
management.sample_retention_policies.basic.minute 	= 5
management.sample_retention_policies.basic.hour   	= 60
management.sample_retention_policies.detailed.10 	= 5

default_user_tags.administrator = true
default_user_tags.management = true

# DISK limits
disk_free_limit.relative = 1.0

# MEM Settings
vm_memory_calculation_strategy = rss
vm_memory_high_watermark.relative = 0.93
vm_memory_high_watermark_paging_ratio = 0.75

# LDAP Settings
auth_backends.1 = internal
auth_backends.2 = ldap
auth_ldap.servers.1  = ldap2012.centrala.test
auth_ldap.user_dn_pattern = CN=${username},CN=Users,DC=centrala,DC=test
auth_ldap.use_ssl    = true
auth_ldap.port       = 636
auth_ldap.log        = false
auth_ldap.timeout    = 60000

# Custom config from databags
cluster_partition_handling = autoheal

Steps to deploy RabbitMQ cluster

RabbitMQ is installed from RHEL REPO
yum install rabbitmq

Steps to reproduce the behavior in question

All three nodes is online.
I stopping node3 by systemctl stop rabbitmq
HTTPS api is currently very slow, about 50sek to display data.
I don't see any warning and problems in RabbitMQ logs.
When I started node3 API is responsive and show data immediately.

advanced.config

See https://www.rabbitmq.com/docs/configure#config-location to learn how to find advanced.config file location

# PASTE advanced.config HERE, BETWEEN BACKTICKS

Application code

[
  {rabbit, [
      {config_entry_decoder, [
             {passphrase,{ file, "/app/rmq/node3/config_pass"} },
             {cipher, blowfish_cfb64},
             {hash, sha256},
             {iterations, 10000}
         ]},
      {credential_validator, [{validation_backend, rabbit_credential_validator_password_regexp},
                                   {regexp, <<"^[a-zA-Z$]{2,99}">>}]
        },
      {default_user, admin},
      {default_pass,
             {encrypted, <<"****************">> }},
      {log,
             [{file,
                 [{count,30},
                  {date,"$D0"},
                  {level,info},
                  {file,"name"}]},
              {categories,
                 [{default,[{level,info}]},
                  {federation,[{level,info}]},
                  {queue,[{file,"queue.log"},{level,info}]},
                  {mirroring,[{file,"mirror.log"},{level,info}]},
                  {channel,[{file,"channel.log"},{level,info}]},
                  {connection,[{file,"connection.log"},{level,info}]}]}]}
    ]},
   {rabbitmq_auth_backend_ldap,[
     {group_lookup_base, "DC=centrala,DC=test"},
     {ssl_options, [
                    {cacertfile,"/etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem"},
                    {verify, verify_peer},
                    {fail_if_no_peer_cert, false}

     ]},
     {vhost_access_query, {'or', [{in_group, "CN=!TEST,OU=Distribution,OU=Groups,OU=_Exchange,DC=centrala,DC=TEST"},
                          {in_group, "CN=!${vhost}_Users,OU=RabbitMQ,OU=Groups,OU=TEST,DC=centrala,DC=TEST"},
                          {in_group, "CN=!${vhost}_Users,OU=RabbitMQ,OU=Groups,OU=TEST,DC=centrala,DC=TEST"}]} },
     {tag_queries, [{administrator, {in_group, "CN=!TEST,OU=Distribution,OU=Groups,OU=_Exchange,DC=centrala,DC=TEST"}},
                    {monitoring, {in_group, "CN=!${vhost}_Users,OU=RabbitMQ,OU=Groups,OU=TEST,DC=centrala,DC=TEST"}},
                    {management, {in_group, "CN=!${vhost}_Users,OU=RabbitMQ,OU=Groups,OU=TEST,DC=centrala,DC=TEST"}}
                    ] }    ]},
    {lager, [
        {error_logger_hwm, 1024}
    ]}
].

Kubernetes deployment file

# Relevant parts of K8S deployment that demonstrate how RabbitMQ is deployed
# PASTE YAML HERE, BETWEEN BACKTICKS

What problem are you trying to solve?

When I started node3 API is responsible and show data immediately.

[mkuratczyk]

1. There were some fixes in this area that I believe were only made in 4.0 (the only version with open source support anyway). Please try on 4.0
2. If you still see the problem - let us know which API endpoint(s) has this problem

[michaelklishin]

HTTPS api is currently very slow, about 50sek to display data

Some endpoints serve 70+ or so metrics, e.g. GET /api/queues. If you try to request all of them without pagination, for 10K queues it will be, say, at least 700K lines in the resulting JSON document.

This and not "API slowness" is usually the key aspect. Use pagination just like the management UI itself does or request metrics of a specific queue.

[michaelklishin]

We won't guess as to what endpoint, and with what kind of topology, is slow for you. Either provide an executable way to reproduce (with obfuscated/dummy data but a comparable number of queues, streams, bindings, and so on), or we won't be able to help even with 4.0.x.

On Linux, our approach to profiling has been documented for years (and before the blog post, elsewhere such as RabbitMQ Summit talks).

[kamilbro5]

Hi, i saw this problem only when i stopped one node with any api endpoint /api/overview /api/nodes/ etc.
also GUI with pagination show website wery slovly.
I saw this problem on cluster with medium traffic (about 100msg/s sum in all queue) and about 50 queue.
CPU utilization was very low, RAM space was free.
AMQP requests worked correctly and quickly

When all 3 nodes is up API and GUI works quickly.

Ok. I will try after upgrade on ver 4.0.

[michaelklishin]

#9522 #9522 #9874