docs: clarify engine to replica ping behavior

[ejweber]

Which issue(s) this PR fixes:

Contributes to longhorn/longhorn#8711.

What this PR does / why we need it:

While working through longhorn/longhorn#8711 (comment), I "discovered" that failure to receive a ping response does NOT cause the engine to mark a replica as ERR. I initially set out to address that "issue".

After quite a bit of internal discussion and POCs that would either remove the engine to replica ping entirely or include it in the rolling engine-to-replica timeout with I/O operations, we have decided to leave it the way it is. See the code comments in this PR for an accurate description of its current behavior.

The main reasons for this decision are:

• The ping can sometimes detect a replica in a weird state (e.g. closed when we think it should be open). It does this in one existing engine integration test. Removing it would potentially reduce our ability to handle an occurrence.
• The ping is otherwise largely only useful when there is no I/O in flight. If there is I/O in flight, engine-replica-timeout governs whether we should mark a replica as ERR.
  • There isn't much benefit to quickly detecting a replica in a bad state if there is no active I/O.
  • If the issue is a temporary network disruption that can resolve itself during a period of no active I/O, it would be quite disruptive (and pointless) to rebuild the replica.

Thanks to @PhanLe1010 for reasoning this out and helping us reach the conclusion.

[PhanLe1010]

LGTM.

Thank you for the investigation