flaky test: can not forward any more: max number of forward reached

[everpcpc]

3: statement failed: mysql client error: Server error: `ERROR HY000 (1105): MetaServiceError. Code: 2001, Text = can not forward any more: max number of forward reached.'
[SQL] insert into t2 values(4,'b_8','c_8');
at tests/sqllogictests/suites/base/09_fuse_engine/./09_0036_merge_into_without_distributed_enable.test:74
at tests/sqllogictests/suites/base/09_fuse_engine/09_0026_merge_into.test:4

https://github.com/databendlabs/databend/actions/runs/11625478696/job/32376074474

[everpcpc]

cc @drmingdrmer could you please help with this?

[drmingdrmer]

It seems the CI machine was under a heavy load, node-1 the leader failed to send a message to node-3 and node-3 start to elect. During this period, there is no Leader and a request can not be forward.

meta-node-3 log:

2024-11-01T07:13:31.922424+00:00[UTC]  INFO openraft::core::raft_core: raft_core.rs:1540 election timeout passed, about to elect
2024-11-01T07:13:31.922481+00:00[UTC]  INFO openraft::core::raft_core: raft_core.rs:1550 do trigger election