Migrate ingesters from single zone to zone-aware replication #9525
Unanswered
tagathi
asked this question in
Help and support
Replies: 0 comments
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
-
Hi,
I am testing the migration of ingesters to zone-aware replication with downtime.
I noticed few things missing from the documented procedure.
I think I have found my way through them, but would appreciate some feedback and maybe suggest some documentation improvements.
First of all, the instructions mention to create an empty migrate.yaml and only paste in it, small parts of configuration. If you do this, the rest of the configuration will take the default values and will mess up your cluster. So what we should really do is take the values.yaml file we use and update it with the suggested configuration.
Then, in step 11 when we start the zone-aware ingesters we need to also set
ingesteblocks_storage.r.ring.zone_awareness_enabled: true
or the installation fails.Finally, what was more confusing for me, was that after completing the migration and enabling the traffic back towards Mimir, my queries did not return any data starting from 12 hours in the past until the moment the new ingesters started runnning.
I had to modify the following parameters to make this work.
querier.query_store_after
querier.query_ingesters_within
blocks_storage.bucket_store.ignore_blocks_within
I set them to the following values:
ignore_blocks_within: 2h
query_store_after: 2h15m
query_ingesters_within: 2h5m
Can you provide some guidance on the values that should be set and possible drawbacks for using such low values?
Beta Was this translation helpful? Give feedback.
All reactions