Document order of replicas is nondeterministic

[nadove-ucsc]

While working on #6122, we had difficulty asserting the contents of the verbatim PFB manifest due to inconsistencies in the ordering of the replicas. The order appeared stable on a personal deployment, but changed when pushing to GitHub. Our investigation revealed that the shard count affected the order of the replicas in the index, but patching the shard count to a consistent value (1) did not result in a consistent order.

Currently, our workaround is to sort the manifests before comparing the expected and observed values.

[nadove-ucsc]

It remains undetermined whether the inconsistency is due to the order in which the replicas are written to the index, or whether it arises when reading them from the index.

[dsotirho-ucsc]

Assignee to consider next steps.

[hannes-ucsc]

We don't currently know what causes the non-determinism. We should probably ask Elastic or the Elasticsearch community.