No DB shards could be opened on single node under heavy parallel inserts

[man4j]

No DB shards could be opened on single node under heavy parallel inserts. If I use q=1 then this error appears earlier, but if I increase q (f.e. q = 8) this error may not appear at all.

Logs:
[error] 2020-12-07T08:06:52.828447Z nonode@nohost <0.20294.2> 86e188ab8c req_err(1108989393) internal_server_error : No DB shards could be opened.
[<<"fabric_util:get_shard/4 L111">>,<<"fabric_util:get_shard/4 L128">>,<<"fabric:get_security/2 L183">>,<<"chttpd_auth_request:db_authorization_check/1 L110">>,<<"chttpd_auth_request:authorize_request/1 L19">>,<<"chttpd:handle_req_after_auth/2 L321">>,<<"chttpd:process_request/1 L306">>,<<"chttpd:handle_request_int/1 L244">>]

Steps to Reproduce

Pseudo-code on Java:

for (int i = 0; i < 60_000; i++) {
MyDoc d = new MyDoc();
d.setContent(doc);
db.async().saveOrUpdate(d).handle((r, t) -> { //ASYNC INSERT!
if (t != null) {
t.printStackTrace();
}
return null;
});
}

Expected Behaviour

Database should work slower and client should get request timeout.

Your Environment

CouchDB 3.1.1, Debian GNU/Linux 10 (buster)

[wohali]

Try increasing [fabric] shard_timeout_factor from 2 up to 10 or more.

[man4j]

Yes, we never use async fire-and-forget requests and always analyze response status, but it's very strange behavior of database. Why database try to process all requests simultaneously without queue? And if queue exceeded limits database should stop process new requests and use tcp connection backlog.

[janl]

patches welcome o/

[man4j]

I found a solution! After changing next parameters, the errors disappeared at all:

[chttpd]
backlog=666666
server_options=[{backlog, 66666}, {acceptor_pool_size, 16}, {max, 66666}]

[couch_httpd_auth]
timeout=66666

[couchdb]
os_process_timeout=66666

[fabric]
shard_timeout_factor=66666

Parameters changed experimentally. And all works great, without error 500.

[lmorandini]

We are experiencing the same behavior during replication of a relatively large (700GB) database with large attachments (a few hundreds of them are bigger than 1GB).

Setting shard_timeout_factor to 10 improved the situation, but the error still occurs intermittently.

@man4j have you changed all the parameters above at once?

[man4j]

I started experimenting recently. And I didn't fully understand what exactly improved the situation. But there are no errors with the above parameters. Other parameters are left as default.

[kocolosk]

A couple of details to keep in mind here. The get_shard routine will cycle through all of the shards in a database trying to find one where it can retrieve the security properties, but it will only examine each shard file once, and it always starts with a short timeout of 100ms. The shard_timeout_factor controls the exponential for the backoff that it uses, e.g. the default of two will cause the system to double the length of the timeout in each iteration, waiting 100ms, 200ms, 400ms, 800ms ... up to the number of shard replicas in your database. If you have q=1 and n=1, then this config setting is irrelevant and the system will fail the request with a get_shard error if it can't open the only shard file in 100ms. That's arguably too aggressive and something we might consider revisiting for that specific configuration.

Conversely, setting a very large timeout factor will cause the system to wait approximately forever trying to open a slow shard file in a database with more than a few shards. That may not be desirable either ...

This get_shard operation should be quite fast when executed on a DB that is already open and processing requests. It only touches in-memory caches of a few bits of metadata. But if the entire server is overloaded it is certainly possible to hit the timeout.