Ingester: err="rpc error: code = Unavailable desc = Starting" msg=gRPC

[zulh-civo]

I'm deploying Cortex using Helm chart with the following Chart values:

alertmanager:
  enabled: false
clusterDomain: cortex.example.com
config:
  auth_enabled: true
  blocks_storage:
    backend: s3
    s3:
      access_key_id: <redacted>
      bucket_name: <redacted>
      endpoint: <redacted>
      region: <redacted>
      secret_access_key: <redacted>
nginx:
  enabled: false
ruler:
  enabled: false
tags:
  blocks-storage-memcached: true
compactor:
  resources:
    requests:
      memory: "64Mi"
      cpu: "250m"
    limits:
      memory: "1Gi"
      cpu: "1000m"
distributor:
  replicas: 3
  resources:
    requests:
      memory: "64Mi"
      cpu: "250m"
    limits:
      memory: "4Gi"
      cpu: "2000m"
  extraArgs:
    "-log.level": "debug"
    "-distributor.ingestion-rate-limit": "500000000"
    # To support `-ingester.max-global-series-per-user` flag
    # "-distributor.shard-by-all-labels": "true"
ingester:
  replicas: 3
  resources:
    requests:
      memory: "64Mi"
      cpu: "250m"
    limits:
      memory: "6Gi"
      cpu: "2000m"
  extraArgs:
    "-log.level": "debug"
    # "-ingester.max-series-per-user": "500000000" 
    # "-ingester.max-global-series-per-user": "500000000"
    "-ingester.ignore-series-limit-for-metric-names": "node_network_transmit_errs_total,apiserver_request_duration_seconds_bucket,container_blkio_device_usage_total,container_tasks_state,node_network_address_assign_type,node_network_carrier,node_network_carrier_changes_total,node_network_carrier_down_changes_total,node_network_carrier_up_changes_total,node_network_device_id,node_network_dormant,node_network_flags,node_network_iface_id,node_network_iface_link,node_network_iface_link_mode,node_network_info,node_network_mtu_bytes,node_network_name_assign_type,node_network_net_dev_group,node_network_protocol_type,node_network_receive_bytes_total,node_network_receive_compressed_total,node_network_receive_drop_total,node_network_receive_errs_total,node_network_receive_fifo_total,node_network_receive_frame_total,node_network_receive_multicast_total,node_network_receive_packets_total,node_network_transmit_bytes_total,node_network_transmit_carrier_total,node_network_transmit_colls_total,node_network_transmit_compressed_total,node_network_transmit_drop_total,node_network_transmit_fifo_total,node_network_transmit_packets_total,node_network_transmit_queue_length,node_network_up,root_ca_cert_publisher_sync_duration_seconds_bucket"
  # statefulSet:
  #   enabled: true
  # persistentVolume:
  #   enabled: true
  #   size: 100Gi
  #   storageClass: ms-xfs-2-replicas
memcached-blocks:
  resources:
    requests:
      memory: "64Mi"
      cpu: "250m"
    limits:
      memory: "128Mi"
      cpu: "500m"
memcached-blocks-index:
  resources:
    requests:
      memory: "64Mi"
      cpu: "250m"
    limits:
      memory: "128Mi"
      cpu: "500m"
memcached-blocks-metadata:
  resources:
    requests:
      memory: "64Mi"
      cpu: "250m"
    limits:
      memory: "128Mi"
      cpu: "500m"
querier:
  resources:
    requests:
      memory: "64Mi"
      cpu: "250m"
    limits:
      memory: "128Mi"
      cpu: "500m"
query_frontend:
  resources:
    requests:
      memory: "64Mi"
      cpu: "250m"
    limits:
      memory: "128Mi"
      cpu: "500m"
store_gateway:
  replicas: 2
  resources:
    requests:
      memory: "64Mi"
      cpu: "250m"
    limits:
      memory: "1Gi"
      cpu: "500m"
runtimeconfigmap:
  runtime_config:
    overrides:
      abc-cluster:
        max_series_per_metric: 500000000
        max_global_series_per_metric: 500000000
        max_series_per_user: 500000000 
        max_global_series_per_user: 500000000 
        max_label_names_per_series: 5000 
      def-cluster:
        max_series_per_metric: 500000000
        max_global_series_per_metric: 500000000
        max_series_per_user: 500000000 
        max_global_series_per_user: 500000000 
        max_label_names_per_series: 5000 
      xyz-cluster:
        max_series_per_metric: 500000000
        max_global_series_per_metric: 500000000
        max_series_per_user: 500000000 
        max_global_series_per_user: 500000000 
        max_label_names_per_series: 5000 

This Cortex is responsible to store all my other cluster metrics where every external cluster/tenant will do remote write and send their metrics to cortex-dist.infra.example.com Ingress which as the 'gateway' for the cortex-distributor Service.

The Ingress is up and running with no issues. The cortex-dist.infra.example.com responds to ping and curl -X POST ... requests.

The external clusters/tenants (abc-cluster, def-cluster and xyz-cluster) are super busy clusters. Each has its own big sets of metrics/timeseries/labels for Cortex to crunch and process...

...which I think leads to the problem that I'm seeing right now. The Ingesters and Distributors are not happy and I'm out of ideas what else I can try to make them happy. I appreciate and welcome all tips/ideas from the community here.

Ingester logs:

cortex-ingester-59c966648b-chff2 ingester level=warn ts=2023-01-25T05:33:44.928744515Z caller=grpc_logging.go:43 method=/cortex.Ingester/Push duration=88.267µs err="rpc error: code = Unavailable desc = Starting" msg=gRPC
cortex-ingester-59c966648b-chff2 ingester level=warn ts=2023-01-25T05:33:44.930058212Z caller=grpc_logging.go:43 method=/cortex.Ingester/Push duration=94.662µs err="rpc error: code = Unavailable desc = Starting" msg=gRPC
cortex-ingester-59c966648b-chff2 ingester level=warn ts=2023-01-25T05:33:44.932207392Z caller=grpc_logging.go:43 method=/cortex.Ingester/Push duration=46.052µs err="rpc error: code = Unavailable desc = Starting" msg=gRPC

Distributor logs:

cortex-distributor-696b474f87-2jwmm distributor level=warn ts=2023-01-25T06:21:20.756667057Z caller=logging.go:86 traceID=27494c8f5fa88f3c msg="POST /api/v1/push (500) 3.519758ms Response: \"DoBatch: InstancesCount <= 0\\n\" ws: false; Content-Encoding: snappy; Content-Length: 54097; Content-Type: application/x-protobuf; User-Agent: Prometheus/2.31.1; X-Forwarded-For: 150.78.11.79; X-Forwarded-Host: cortex-dist.infra.example.com; X-Forwarded-Port: 443; X-Forwarded-Proto: https; X-Forwarded-Scheme: https; X-Prometheus-Remote-Write-Version: 0.1.0; X-Real-Ip: 150.78.11.79; X-Request-Id: 831500282a540d5bf582c9e3bcf2ba77; X-Scheme: https; X-Scope-Orgid: abc-cluster; "
cortex-distributor-696b474f87-dcl8f distributor level=warn ts=2023-01-25T06:21:02.780356803Z caller=logging.go:86 traceID=317f94b8e8bfbfb6 msg="POST /api/v1/push (500) 3.85498ms Response: \"at least 2 live replicas required, could only find 1 - unhealthy instances: 10.244.8.73:9095,10.244.2.49:9095\\n\" ws: false; Content-Encoding: snappy; Content-Length: 17920; Content-Type: application/x-protobuf; User-Agent: Prometheus/2.31.1; X-Forwarded-For: 26.174.48.133; X-Forwarded-Host: cortex-dist.infra.example.com; X-Forwarded-Port: 443; X-Forwarded-Proto: https; X-Forwarded-Scheme: https; X-Prometheus-Remote-Write-Version: 0.1.0; X-Real-Ip: 26.174.48.133; X-Request-Id: acfa5cf314b9e65cfcaf00b54b2589ff; X-Scheme: https; X-Scope-Orgid: def-cluster; "
cortex-distributor-696b474f87-dcl8f distributor level=warn ts=2023-01-25T06:21:02.780558082Z caller=logging.go:86 traceID=2754fe58221bbec2 msg="POST /api/v1/push (500) 2.471467ms Response: \"at least 2 live replicas required, could only find 1 - unhealthy instances: 10.244.8.73:9095,10.244.2.49:9095\\n\" ws: false; Content-Encoding: snappy; Content-Length: 33254; Content-Type: application/x-protobuf; User-Agent: Prometheus/2.31.1; X-Forwarded-For: 234.159.208.213; X-Forwarded-Host: cortex-dist.infra.example.com; X-Forwarded-Port: 443; X-Forwarded-Proto: https; X-Forwarded-Scheme: https; X-Prometheus-Remote-Write-Version: 0.1.0; X-Real-Ip: 234.159.208.213; X-Request-Id: 5efefaf0c1c28fda316cc50cd708de4e; X-Scheme: https; X-Scope-Orgid: xyz-cluster; "

Prometheus logs from one of the tenants (abc-cluster in this case):

prometheus-prom-kube-prometheus-stack-prometheus-0 prometheus ts=2023-01-25T05:30:17.740Z caller=dedupe.go:112 component=remote level=warn remote_name=ed7512 url=https://cortex-dist.infra.example.com/api/v1/push msg="Failed to send batch, retrying" err="server returned HTTP status 500 Internal Server Error: rpc error: code = Unavailable desc = error reading from server: read tcp 10.244.8.71:52740->10.244.8.73:9095: use of closed network connection"
prometheus-prom-kube-prometheus-stack-prometheus-0 prometheus ts=2023-01-25T05:30:18.527Z caller=dedupe.go:112 component=remote level=warn remote_name=ed7512 url=https://cortex-dist.infra.example.com/api/v1/push msg="Failed to send batch, retrying" err="server returned HTTP status 500 Internal Server Error: rpc error: code = Unavailable desc = error reading from server: read tcp 10.244.9.225:50322->10.244.8.73:9095: use of closed network connection"

Any ideas?

[alanprot]

Are the ingesters jn a crash loop? They should stay in starting only while replaying the wall

[zulh-civo]

Yes, they are crashing:

$ kgp -l=app.kubernetes.io/component=ingester,app.kubernetes.io/instance=cortex
NAME                               READY   STATUS             RESTARTS   AGE
cortex-ingester-59c966648b-67m24   0/1     CrashLoopBackOff   27         3h9m
cortex-ingester-59c966648b-chff2   0/1     CrashLoopBackOff   28         3h9m
cortex-ingester-59c966648b-f59hq   0/1     CrashLoopBackOff   26         152m

Events from the cortex-ingester-59c966648b-67m24 pod:

Events:
  Type     Reason     Age                     From     Message
  ----     ------     ----                    ----     -------
  Warning  Unhealthy  34m (x395 over 3h9m)    kubelet  Readiness probe failed: HTTP probe failed with statuscode: 503
  Warning  BackOff    4m45s (x512 over 150m)  kubelet  Back-off restarting failed container

[zulh-civo]

@alanprot - how can I make the probes happy?

[alanprot]

I don’t think the probe is the reason why is crashing .. if u describe the pod you can see (terminate reason j think) …

also ingesters should be statefulsets in most of the cases and seems is a deployment in your case

[zulh-civo]

Thanks, I'll try statefulsets.

I described the pod and saw this:

    State:          Waiting
      Reason:       CrashLoopBackOff
    Last State:     Terminated
      Reason:       OOMKilled
      Exit Code:    137
      Started:      Wed, 25 Jan 2023 16:34:41 +0800
      Finished:     Wed, 25 Jan 2023 16:35:37 +0800
    Ready:          False
    Restart Count:  34

I think it crashed due to OOM -> restart -> OOM again and again

[friedrichg]

Configure your ingester resources like this:

    requests:
      memory: "6Gi"
      cpu: "2000m"
    limits:
      memory: "6Gi"
      cpu: "2000m"

if you specify less requests than limits, you can run into resources issues. This is standard for all pods in kubernetes

[zulh-civo]

I'm changing the Ingester from Deployment to Statefulset (as per @alanprot's suggestion) with 100GB volume each. I also increased the requests and limits, making them same values (as per @friedrichg's suggestion).

My Helm values now look like this:

alertmanager:
  enabled: false
clusterDomain: cortex.example.com
config:
  auth_enabled: true
  blocks_storage:
    backend: s3
    s3:
      access_key_id: <redacted>
      bucket_name: <redacted>
      endpoint: <redacted>
      region: <redacted>
      secret_access_key: <redacted>
nginx:
  enabled: false
ruler:
  enabled: false
tags:
  blocks-storage-memcached: true
compactor:
  resources:
    requests:
      memory: "1Gi"
      cpu: "1000m"
    limits:
      memory: "1Gi"
      cpu: "1000m"
distributor:
  replicas: 3
  resources:
    requests:
      memory: "4Gi"
      cpu: "2000m"
    limits:
      memory: "4Gi"
      cpu: "2000m"
  extraArgs:
    "-log.level": "debug"
    "-distributor.ingestion-rate-limit": "500000000"
    # To support `-ingester.max-global-series-per-user` flag
    # "-distributor.shard-by-all-labels": "true"
ingester:
  replicas: 3
  resources:
    requests:
      memory: "35Gi"
      cpu: "4000m"
    limits:
      memory: "35Gi"
      cpu: "4000m"
  extraArgs:
    "-log.level": "debug"
    # "-ingester.max-series-per-user": "500000000" 
    # "-ingester.max-global-series-per-user": "500000000"
    "-ingester.ignore-series-limit-for-metric-names": "node_network_transmit_errs_total,apiserver_request_duration_seconds_bucket,container_blkio_device_usage_total,container_tasks_state,node_network_address_assign_type,node_network_carrier,node_network_carrier_changes_total,node_network_carrier_down_changes_total,node_network_carrier_up_changes_total,node_network_device_id,node_network_dormant,node_network_flags,node_network_iface_id,node_network_iface_link,node_network_iface_link_mode,node_network_info,node_network_mtu_bytes,node_network_name_assign_type,node_network_net_dev_group,node_network_protocol_type,node_network_receive_bytes_total,node_network_receive_compressed_total,node_network_receive_drop_total,node_network_receive_errs_total,node_network_receive_fifo_total,node_network_receive_frame_total,node_network_receive_multicast_total,node_network_receive_packets_total,node_network_transmit_bytes_total,node_network_transmit_carrier_total,node_network_transmit_colls_total,node_network_transmit_compressed_total,node_network_transmit_drop_total,node_network_transmit_fifo_total,node_network_transmit_packets_total,node_network_transmit_queue_length,node_network_up,root_ca_cert_publisher_sync_duration_seconds_bucket"
  statefulSet:
    enabled: true
  persistentVolume:
    enabled: true
    size: 100Gi
    storageClass: ms-xfs-2-replicas
memcached-blocks:
  resources:
    requests:
      memory: "128Mi"
      cpu: "500m"
    limits:
      memory: "128Mi"
      cpu: "500m"
memcached-blocks-index:
  resources:
    requests:
      memory: "128Mi"
      cpu: "500m"
    limits:
      memory: "128Mi"
      cpu: "500m"
memcached-blocks-metadata:
  resources:
    requests:
      memory: "128Mi"
      cpu: "500m"
    limits:
      memory: "128Mi"
      cpu: "500m"
querier:
  resources:
    requests:
      memory: "128Mi"
      cpu: "500m"
    limits:
      memory: "128Mi"
      cpu: "500m"
query_frontend:
  resources:
    requests:
      memory: "128Mi"
      cpu: "500m"
    limits:
      memory: "128Mi"
      cpu: "500m"
store_gateway:
  replicas: 2
  resources:
    requests:
      memory: "1Gi"
      cpu: "500m"
    limits:
      memory: "1Gi"
      cpu: "500m"
runtimeconfigmap:
  runtime_config:
    overrides:
      abc-cluster:
        max_series_per_metric: 500000000
        max_global_series_per_metric: 500000000
        max_series_per_user: 500000000 
        max_global_series_per_user: 500000000 
        max_label_names_per_series: 5000 
      def-cluster:
        max_series_per_metric: 500000000
        max_global_series_per_metric: 500000000
        max_series_per_user: 500000000 
        max_global_series_per_user: 500000000 
        max_label_names_per_series: 5000 
      xyz-cluster:
        max_series_per_metric: 500000000
        max_global_series_per_metric: 500000000
        max_series_per_user: 500000000 
        max_global_series_per_user: 500000000 
        max_label_names_per_series: 5000 

While I don't see err="rpc error: code = Unavailable desc = Starting" msg=gRPC error anymore, I'm seeing different errors now:

cortex-ingester-0 ingester level=debug ts=2023-01-30T09:44:37.918443964Z caller=grpc_logging.go:46 method=/cortex.Ingester/Push duration=10.250195ms msg="gRPC (success)"
cortex-ingester-0 ingester level=debug ts=2023-01-30T09:44:37.920031602Z caller=grpc_logging.go:46 method=/cortex.Ingester/Push duration=14.202785ms msg="gRPC (success)"
cortex-ingester-0 ingester level=debug ts=2023-01-30T09:44:37.921327087Z caller=grpc_logging.go:46 method=/cortex.Ingester/Push duration=12.430144ms msg="gRPC (success)"
cortex-ingester-0 ingester level=debug ts=2023-01-30T09:44:37.922281677Z caller=grpc_logging.go:46 method=/cortex.Ingester/Push duration=7.81947ms msg="gRPC (success)"
cortex-ingester-0 ingester level=debug ts=2023-01-30T09:44:37.922730215Z caller=grpc_logging.go:46 method=/cortex.Ingester/Push duration=3.102237ms msg="gRPC (success)"
cortex-ingester-2 ingester level=warn ts=2023-01-30T09:44:37.918416888Z caller=grpc_logging.go:43 method=/cortex.Ingester/Push duration=2.844779ms err="user=def-cluster-supercluster: write to WAL: log samples: write /data/tsdb/def-cluster-supercluster/wal/00000032: input/output error" msg=gRPC
cortex-ingester-2 ingester level=warn ts=2023-01-30T09:44:37.91885326Z caller=grpc_logging.go:43 method=/cortex.Ingester/Push duration=3.282156ms err="user=abc-cluster-supercluster: write to WAL: log samples: write /data/tsdb/abc-cluster-supercluster/wal/00000046: input/output error" msg=gRPC
cortex-ingester-2 ingester level=warn ts=2023-01-30T09:44:37.918857187Z caller=grpc_logging.go:43 method=/cortex.Ingester/Push duration=2.93359ms err="user=def-cluster-supercluster: write to WAL: log samples: write /data/tsdb/def-cluster-supercluster/wal/00000032: input/output error" msg=gRPC
cortex-ingester-2 ingester level=warn ts=2023-01-30T09:44:37.919330928Z caller=grpc_logging.go:43 method=/cortex.Ingester/Push duration=3.361923ms err="user=abc-cluster-supercluster: write to WAL: log samples: write /data/tsdb/abc-cluster-supercluster/wal/00000046: input/output error" msg=gRPC
cortex-ingester-2 ingester level=warn ts=2023-01-30T09:44:37.919894926Z caller=grpc_logging.go:43 method=/cortex.Ingester/Push duration=3.506042ms err="user=def-cluster-supercluster: write to WAL: log samples: write /data/tsdb/def-cluster-supercluster/wal/00000032: input/output error" msg=gRPC
cortex-ingester-0 ingester level=debug ts=2023-01-30T09:44:37.923021238Z caller=grpc_logging.go:46 method=/cortex.Ingester/Push duration=8.441769ms msg="gRPC (success)"
cortex-ingester-0 ingester level=debug ts=2023-01-30T09:44:37.923182559Z caller=grpc_logging.go:46 method=/cortex.Ingester/Push duration=4.349758ms msg="gRPC (success)"

Any idea?

[friedrichg]

If original issue is gone, you should close this discussion and open a new one.
From the logs you shared, the warnings point to problems writing to the disk attached to the ingester. Most likely not related to cortex directly. Ensure you have enough space to write in the disks.