Questions on ingester on Kubernetes setup

[nosalan]

Please don't mind some beginners questions

1. Is it possible to run microservice mode/non zonally-aware with just 1 ingester? I don't care about potential lack of HA.
2. As I understand the purpose of ingester statefulset is to provide HA because each replica has roughy the same data. Should I increase the amount of replicas in a statefulset also when the load gets higher? Or generally what are the scaling guidelines for ingesters?
3. Which settings can I change so that data is flushed to cloud (Azure storage) as soon as possible and removed from local ingesters storage as soon as possible?
4. What can be done to decrease single ingester instance memory usage.

[dimitarvdimitrov]

• Is it possible to run microservice mode/non zonally-aware with just 1 ingester? I don't care about potential lack of HA.
  yes, that's possible. You can set -ingester.ring.replication-factor=1

• As I understand the purpose of ingester statefulset is to provide HA because each replica has roughy the same data. Should I increase the amount of replicas in a statefulset also when the load gets higher? Or generally what are the scaling guidelines for ingesters?

Replicas in a statefulset split the data and replicate it based on the -ingester.ring.replication-factor config option. You can add more replicas to the statefulset to spread out load.

• Which settings can I change so that data is flushed to cloud (Azure storage) as soon as possible and removed from local ingesters storage as soon as possible?

I think these should be sufficient. You should still leave about 4-5 hours between data being ingested and being shipped to object store and discoverable. The minimum duration to wait is 3 hours as that data is still in ingesters' memory.

• -querier.query-ingesters-within
• -querier.query-store-after
• -blocks-storage.tsdb.retention-period
• -blocks-storage.bucket-store.ignore-blocks-within

• What can be done to decrease single ingester instance memory usage.

Perhaps trying the options above is a good starting point.