avalanchego out of file descriptors (too many open files)

[borutmrak]

Hello,

We're running an FLR node with this configuration:

{
  "snowman-api-enabled": false,
  "coreth-admin-api-enabled": false,
  "pruning-enabled": false,
  "eth-apis": [
    "eth",
    "eth-filter",
    "net",
    "web3",
    "internal-eth",
    "internal-blockchain",
    "internal-transaction",
    "debug-tracer"
  ],
  "metrics-enabled": true,
  "metrics-expensive-enabled": false,
  "fd-limit": 100000
}

We're running docker image flarefoundation/go-flare:latest (9f816a797487) inside an LXC container.

This is the only node we have with debug-tracer enabled and it was exhibiting this behaviour:

C-2024-01-05T09-59-34.877.log:[01-04|02:32:46.772] FATAL <C Chain> handler/handler.go:282 shutting down chain {"reason": "received an unexpected error", "error": "failed to load index checkpoint: closed to avoid possible corruption, init error: open /app/db/flare/v1.4.5/39424475.ldb: too many open files while processing sync message: (Op: put, NodeID: NodeID-MfzBQ6MbriE31UWm9NjSnEgLiR1cAkEnN, RequestID: 285091, ContainerID: 0x541d7629442119bb498984e26620866ec2ce96aa982a5e5385134d7c63e7fd0a)"}
C-2024-01-05T09-59-34.877.log:[01-04|02:32:46.772] ERROR <C Chain> handler/handler.go:775 failed while shutting down the chain {"error": "closed to avoid possible corruption, init error: open /app/db/flare/v1.4.5/39424475.ldb: too many open files"}
C-2024-01-05T09-59-34.877.log:WARN [01-04|16:26:32.723] <C Chain> github.com/ava-labs/coreth/core/rawdb/accessors_metadata.go:157: Error reading unclean shutdown markers error="closed to avoid possible corruption, init error: open /app/db/flare/v1.4.5/39518175.ldb: too many open files"
C-2024-01-05T09-59-34.877.log:WARN [01-04|16:31:32.722] <C Chain> github.com/ava-labs/coreth/core/rawdb/accessors_metadata.go:157: Error reading unclean shutdown markers error="closed to avoid possible corruption, init error: open /app/db/flare/v1.4.5/39518175.ldb: too many open files"
C-2024-01-05T09-59-34.877.log:WARN [01-04|16:36:32.722] <C Chain> github.com/ava-labs/coreth/core/rawdb/accessors_metadata.go:157: Error reading unclean shutdown markers error="closed to avoid possible corruption, init error: open /app/db/flare/v1.4.5/39518175.ldb: too many open files"
C-2024-01-05T09-59-34.877.log:WARN [01-04|16:41:32.722] <C Chain> github.com/ava-labs/coreth/core/rawdb/accessors_metadata.go:157: Error reading unclean shutdown markers error="closed to avoid possible corruption, init error: open /app/db/flare/v1.4.5/39518175.ldb: too many open files"
C-2024-01-05T09-59-34.877.log:[01-04|16:45:55.611] ERROR <C Chain> handler/handler.go:775 failed while shutting down the chain {"error": "closed to avoid possible corruption, init error: open /app/db/flare/v1.4.5/39518175.ldb: too many open files"}

After some research I found out that the default file descriptor limit in Avalanche is 32768 (https://docs.avax.network/nodes/configure/avalanchego-config-flags#file-descriptor-limit) and can be set by the command line option --fd-limit (or as above in the node config). Setting it to 100K resolved the problem.

The node itself is just available to a few clients and does not do much else, however we don't see this error on other, much more used nodes.

I think the default FD limit should be higher or it should at least be documented that this issue might arise for users. At first glance it looked like leveldb corruption - which it luckily isn't.

best regards,
B.