"per-user series limit of 150000 exceeded" in mimir-distributed Helm chat

[cyrilbkr]

Hi

I'm testing Mimir on a large scale environment and use the mimir-distributed Helm chart following the documentation.
I am reaching the default limit on the distributor. It's not really clear in the doc where it's possible to tweak the value In the values.yaml of the Helm chart.

Error in distributor logs :
user=anonymous msg="push error" err="rpc error: code = Code(400) desc = user=anonymous: per-user series limit of 150000 exceeded (err-mimir-max-series-per-user). To adjust the related per-tenant limit, configure -ingester.max-global-series-per-user, or contact your service administrator."

To Reproduce

I try to setup that In my custom.yaml but I am still having the same error.

...
ingester_limits:
  max_global_series_per_user: 2000000
  max_series: 5000000
...

Chart version : mimir-distributed-3.0.0

[pracucci]

Most common errors have a ID you can lookup in our runbooks (or search on Google). In your case the ID is err-mimir-max-series-per-user and the runbook is:
https://grafana.com/docs/mimir/latest/operators-guide/mimir-runbooks/#err-mimir-max-series-per-user

About your specific question on how to configure it the limit using YAML, the configuration file reference doc is here:
https://grafana.com/docs/mimir/latest/operators-guide/configure/reference-configuration-parameters/

In particular, the correct YAML configuration snippet is (see limits reference doc):

limits:
  max_global_series_per_user: 2000000

[dimitarvdimitrov]

to use this yaml snippet in the helm chart, you can do something like

runtimeConfig:
  ingester_limits:
    max_global_series_per_user: 2000000

or

mimir:
  structuredConfig:
    limits:
      max_global_series_per_user: 2000000

having it in the runtime config has the benefit that the helm upgrade will not restart all components. But both cases achieve the same

[pracucci]

@dimitarvdimitrov Is it documented anywhere in the mimir doc? I would love to link it from our runbooks and config doc.

[dimitarvdimitrov]

unfortunately it isn't. There is https://grafana.com/docs/mimir/latest/operators-guide/running-production-environment-with-helm/configuration-with-helm/, but it doesn't cover runtime config. I will create an issue to add it and update the runbooks to mention helm and how to set the settings in helm.

edit: opened #3135

[bebosudo]

We just had the same issue: I tried setting the value in runtimeConfig but it made the ingestors fail with the following error:

ts=2024-10-18T14:56:52.626378447Z caller=mimir.go:902 level=error msg="module failed" module=runtime-config err="starting module runtime-config: invalid service state: Failed, expected: Running, failure: failed to load runtime config: load file: yaml: unmarshal errors:\n  line 2: field max_global_series_per_user not found in type ingester.plain"

Setting the value in mimir.structuredConfig.limits worked fine instead.

[cyrilbkr]

The current documentation doesn't cover the runtime configuration for Helm, so it's not really clear how to custom the value in the chart.

That's what I was looking for @dimitarvdimitrov !

Thanks a lot guys !

[algo7]

The doc is still not there for Helm Charts almost 2 years later. But if you check the general configuration for Mimir it's actually there: https://grafana.com/docs/mimir/latest/configure/configuration-parameters/