How to prevent Querier crashes?

[weeco]

I noticed some user queries lead to querier crashes (OOM killed I think). Since we already slice the ingoing query into daily slices I wonder what can cause the queries to be so expensive that they can crash a querier that always has plenty of RAM available (2-4Gb).

What options do I have to prevent these querier crashes? Looking at the number of restarts they seem to happen like once a day only and the query seem to succeed after a few retries. So this is not an actual problem but I'm interested in what's going on there and how I could possibly improve the situation.

[pracucci]

Some general suggestions:

• Give queriers more memory (eg. we run queriers with 25GB memory limit, but we also handle pretty heavy queries)
• Fine tune query-related limits
  • -store.query-chunk-limit
  • -store.max-query-length
  • -querier.max-query-parallelism