No DB shards could be opened - req_err(1577439538)

[BrodaUa]

Description

I try to build a messaging app with CouchDB +NodeJS using nano lib. For that I try to create a large number of user-message dbs (up to 10k) and start listening changes feed of each db to track new message and reroute it accordingly.
Currently, when I try to create 10k of dbs and subscribe to the I get No DB shards could be opened - please, see attached error log. This issue linked to the ticket in couchdb-nano repo: apache/couchdb-nano#267 . In short, before I had a performance issue that nano was picking up last updates very slowly. Following the advice to increase maxSockets of http agent, now I am stuck with No DB shards could be opened issue.

I tried to increase the file descriptors value of the docker container to 128k, but still have the issue.

Steps to Reproduce

1. create 10k databases
2. try to subscribe to changes feed

Expected Behaviour

no errors

Your Environment

• CouchDB version used: couchdb:3.1 from https://hub.docker.com/_/couchdb/
• Browser name and version:
• Operating system and version: ubuntu 18.04

Additional Context

App error message:

{"message":" No DB shards could be opened.","level":"error","timestamp":"2021-05-19T12:08:48.160Z","metadata":{"scope":"couch","statusCode":500,"request":{"method":"post","headers":{"content-type":"application/json","accept":"application/json","user-agent":"nano/9.0.3 (Node.js v12.22.1)","Accept-Encoding":"deflate, gzip"},"agent":null,"qsStringifyOptions":{"arrayFormat":"repeat"},"url":"http://XXXXXX:XXXXXX@127.0.0.1:5984/sampleDB/_changes","params":{"feed":"longpoll","timeout":60000,"since":"now","limit":100,"include_docs":true},"data":"{}","maxRedirects":0,"httpAgent":null,"httpsAgent":null},"headers":{"uri":"http://XXXXXX:XXXXXX@127.0.0.1:5984/sampleDB/_changes","statusCode":500,"cache-control":"must-revalidate","connection":"close","content-type":"application/json","date":"Wed, 19 May 2021 12:06:07 GMT","x-couch-request-id":"6165cd9925","x-couch-stack-hash":"1577439538","x-couchdb-body-time":"0"},"errid":"non_200","name":"Error","description":"No DB shards could be opened.","error":"internal_server_error","reason":"No DB shards could be opened.","ref":1577439538,"stack":"Error: No DB shards could be opened.\n at responseHandler (/home/user/project/node_modules/nano/lib/nano.js:175:20)\n at /home/user/project/node_modules/nano/lib/nano.js:405:13\n at runMicrotasks (<anonymous>)\n at processTicksAndRejections (internal/process/task_queues.js:97:5)"}}

CouchDB error:

[error] 2021-05-19T12:10:00.576437Z nonode@nohost <0.17889.57> 95d02fadcb req_err(1577439538) internal_server_error : No DB shards could be opened.
[<<"fabric_util:get_shard/4 L111">>,<<"fabric_util:get_shard/4 L128">>,<<"fabric_util:get_shard/4 L128">>,<<"fabric:get_security/2 L183">>,<<"chttpd_auth_request:db_authorization_check/1 L110">>,<<"chttpd_auth_request:authorize_request/1 L19">>,<<"chttpd:handle_req_after_auth/2 L321">>,<<"chttpd:process_request/1 L306">>]

linked issue apache/couchdb-nano#267

[KorbinianKuhn]

I have the same problem with a different setup:

• CouchDB with db per user (>20000 users)
• NodeJS server as a proxy between CouchDB and Client
• Clientside PouchDBs that have live-replication to their Users-CouchDB and subscribe to the changes feed

[nickva]

This issue is usually more pronounced in dbs with lower q and n values, what q and n values do you have for those dbs?

[KorbinianKuhn]

I have a single node cluster, so I set q=1, n=1. But it has no effect anyway right?

[nickva]

PR with a fix was just merged #3734 should be available in the next version (3.2)

[nickva]

In the meantime it is possible to adjust the [fabric] shard_timeout_factor to 10 or 15 and it should help with the low q/n dbs

[KorbinianKuhn]

I set the shard_timeout_factor to 15, but I still get many errors (2k in the past 6 hours): req_err(1108989393) or req_err(1577439538).

Should I open another issue or wait for v3.2?

[nickva]

Ah, with q=1,n=1 the shard_timeout_factor might not help, as there is only one shard to try and the timeout factor would take effect only after the first failure. In that case might be best to wait for 3.2

[KorbinianKuhn]

For anyone having the same issues: After digging deeper into our monitoring stats, I realized that the disk utilization was very high. We switched from an HDD-Raid to SSD and now it's way better. We still get some errors, but only a few. Looking forward to v3.2, maybe then we can avoid them completely.