[Feature Request] Implement mechanism to fail stale search replicas

[vinaykpud]

Is your feature request related to a problem? Please describe

Currently, there is no mechanism in place to automatically fail search replicas that are significantly lagging behind the primary shard and have become stale. This can lead to inconsistent search results by returning stale data.

Describe the solution you'd like

With Issue #16801, we are proposing to redefine the computation of lag. The lag will now be defined as the difference between the current time and the timestamp of the latest received checkpoint (cp). This change means we will no longer compare the lag with the primary shard directly. Instead, we need a mechanism to monitor the lag in search replica shards. If a search replica exceeds a predefined lag threshold, it should be marked as stale and automatically fail.

Related component

Search:Performance

Describe alternatives you've considered

No response

Additional context

No response

[sandeshkr419]

[Search Triage] - I think this might better fit in with index/segment replication side. Adding those tags and keeping untriaged label.

[vinaykpud]

This is related to Meta issue : #15306