External MinIO doesn´t receive blocks from GrafanaMimir

[MarcelGlaub]

I have installed a GrafanaMimir on a Kubernetes cluster and a MinIO on a VM. I want Mimir to send the metrics/blocks to the MinIO on the VM. Apart from the file "mimir_cluster_seed.json" in the bucket mimir-tsdb all buckets are empty.

I can't see any errors in the pod logs. Previously I used the MinIO built into GrafanaMimir and was able to retrieve all metrics.
I am using the following values:

values.txt

I would be grateful for tips on where else I can look or what I am doing wrong

[dimitarvdimitrov]

it can take up to 3 hours for the metrics to show in the bucket. Metrics are kept on the ingester memory for up to 3 hours after reception.

[MarcelGlaub]

Hi, Mimir has been running with this config for 4 hours already. I have also tried this over the last few days with various configs and didnt get it to work. Pretty sure im doing something wrong

[dimitarvdimitrov]

the values.txt look ok to me. The large retention in the tsdb block should only apply to after shipping the blocks, not before. Do you find any error logs in the ingester (should contain the err string)?

Stupid question, but something is sending metrics to the cluster already, right?

[MarcelGlaub]

The only error I can see is this one:
caller=resolver.go:87 level=error msg="failed to lookup IP addresses" host=mimir-gossip-ring.monitoring.svc.cluster.local err="lookup mimir-gossip-ring.monitoring.svc.cluster.local on 10.43.0.10:53: no such host"

But it is followed by these two that tell me everything is ok as far as I understand it:
caller=memberlist_client.go:595 level=info msg="joining memberlist cluster succeeded" reached_nodes=12 elapsed_time=39.883669597s
caller=seed.go:141 level=info msg="found a cluster seed file created recently, waiting until stable"

But maybe the top error actually is my Problem then?

I have attached the log if you want to have a look yourself.
ingester_log.txt

Yes, a Prometheus sends metrics to Mimir via remote write. I can also visualise the metrics in Grafana if I use the built-in minio instead of the MinIO VM.

[MarcelGlaub]

Update: I fixed my above gossip ring/memberlist error and logs are looking very good now as far as i can tell:

`ts=2024-01-15T10:48:44.387528152Z caller=main.go:216 level=info msg="Starting application" version="(version=2.9.0, branch=HEAD, revision=761114d)"
ts=2024-01-15T10:48:44.388894618Z caller=server.go:336 level=info http=[::]:8080 grpc=[::]:9095 msg="server listening on addresses"
ts=2024-01-15T10:48:44.395602989Z caller=ingester.go:335 level=info msg="TSDB idle compaction timeout set" timeout=1h3m15.558655748s
ts=2024-01-15T10:48:44.396517666Z caller=memberlist_client.go:437 level=info msg="Using memberlist cluster label and node name" cluster_label= node=mimir-ingester-zone-a-0-35e3be48
ts=2024-01-15T10:48:44.39775069Z caller=module_service.go:82 level=info msg=initialising module=sanity-check
ts=2024-01-15T10:48:44.397924501Z caller=sanity_check.go:32 level=info msg="Checking directories read/write access"
ts=2024-01-15T10:48:44.399005018Z caller=module_service.go:82 level=info msg=initialising module=usage-stats
ts=2024-01-15T10:48:44.399902561Z caller=module_service.go:82 level=info msg=initialising module=active-groups-cleanup-service
ts=2024-01-15T10:48:44.400054061Z caller=module_service.go:82 level=info msg=initialising module=activity-tracker
ts=2024-01-15T10:48:44.401481663Z caller=sanity_check.go:37 level=info msg="Directories read/write access successfully checked"
ts=2024-01-15T10:48:44.401581857Z caller=sanity_check.go:39 level=info msg="Checking object storage config"
ts=2024-01-15T10:48:44.402302097Z caller=reporter.go:133 level=info msg="usage stats reporter initialized" cluster_id=14eb4edb-bea5-41c4-b2ad-153563542dfc
ts=2024-01-15T10:48:44.405571478Z caller=sanity_check.go:44 level=info msg="Object storage config successfully checked"
ts=2024-01-15T10:48:44.405602617Z caller=module_service.go:82 level=info msg=initialising module=server
ts=2024-01-15T10:48:44.405698222Z caller=module_service.go:82 level=info msg=initialising module=memberlist-kv
ts=2024-01-15T10:48:44.405720373Z caller=module_service.go:82 level=info msg=initialising module=runtime-config
ts=2024-01-15T10:48:44.405954581Z caller=module_service.go:82 level=info msg=initialising module=ingester-service
ts=2024-01-15T10:48:44.405978018Z caller=ingester.go:2052 level=info msg="opening existing TSDBs"
ts=2024-01-15T10:48:44.406408431Z caller=mimir.go:825 level=info msg="Application started"
ts=2024-01-15T10:48:44.419274923Z caller=memberlist_client.go:543 level=info msg="memberlist fast-join starting" nodes_found=26 to_join=8
ts=2024-01-15T10:48:44.429543429Z caller=memberlist_client.go:563 level=info msg="memberlist fast-join finished" joined_nodes=8 elapsed_time=31.409196ms
ts=2024-01-15T10:48:44.429616399Z caller=memberlist_client.go:576 level=info msg="joining memberlist cluster" join_members=dnssrv+grafana-mimir-headless.monitoring.svc.cluster.local:7946
ts=2024-01-15T10:48:44.429698667Z caller=lifecycler.go:589 level=info msg="adding tokens from file" num_tokens=512
ts=2024-01-15T10:48:44.525123957Z caller=memberlist_client.go:595 level=info msg="joining memberlist cluster succeeded" reached_nodes=16 elapsed_time=95.507546ms`

I will now wait the 4 hours to see if blocks arrive in the buckets

[MarcelGlaub]

Update again:
for some reason the error didnt show up in the logs yesterday. Today i redeployed mimir since there were no blocks in the minio buckets and the error comes up again.
caller=resolver.go:87 level=error msg="failed to lookup IP addresses" host=mimir-gossip-ring.monitoring.svc.cluster.local err="lookup mimir-gossip-ring.monitoring.svc.cluster.local on 10.43.0.10:53: no such host"

but when I run nslookup mimir-gossip-ring.monitoring.svc.cluster.local on one pod I get all the IP's of the other pods back as expected:

$ nslookup mimir-gossip-ring.monitoring.svc.cluster.local
Server:         10.43.0.10
Address:        10.43.0.10:53

Name:   mimir-gossip-ring.monitoring.svc.cluster.local
Address: 10.42.2.168
Name:   mimir-gossip-ring.monitoring.svc.cluster.local
Address: 10.42.4.89
Name:   mimir-gossip-ring.monitoring.svc.cluster.local
Address: 10.42.4.88
Name:   mimir-gossip-ring.monitoring.svc.cluster.local
Address: 10.42.4.95
Name:   mimir-gossip-ring.monitoring.svc.cluster.local
Address: 10.42.3.253
Name:   mimir-gossip-ring.monitoring.svc.cluster.local
Address: 10.42.2.170
Name:   mimir-gossip-ring.monitoring.svc.cluster.local
Address: 10.42.2.169
Name:   mimir-gossip-ring.monitoring.svc.cluster.local
Address: 10.42.4.92
Name:   mimir-gossip-ring.monitoring.svc.cluster.local
Address: 10.42.3.254
Name:   mimir-gossip-ring.monitoring.svc.cluster.local
Address: 10.42.2.164
Name:   mimir-gossip-ring.monitoring.svc.cluster.local
Address: 10.42.4.94
Name:   mimir-gossip-ring.monitoring.svc.cluster.local
Address: 10.42.3.252

Any ideas?