Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Error in alloy-logs when using k8s-monitoring-helm v2 #959

Closed
cxk314 opened this issue Nov 27, 2024 · 6 comments
Closed

Error in alloy-logs when using k8s-monitoring-helm v2 #959

cxk314 opened this issue Nov 27, 2024 · 6 comments

Comments

@cxk314
Copy link

cxk314 commented Nov 27, 2024

Hi,
I am getting following error in Alloy Logs when using k8s-monitoring-helm v2 - 2.0.0-rc.5

ts=2024-11-27T02:30:05.980421464Z level=error msg="failed to consume log entries" component_path=/ component_id=otelcol.receiver.loki.otlp err="sending queue is full"
ts=2024-11-27T02:30:05.980435306Z level=error msg="Exporting failed. Rejecting data." component_path=/ component_id=otelcol.exporter.otlp.otlp error="sending queue is full" rejected_items=1
ts=2024-11-27T02:30:05.980440148Z level=error msg="failed to consume log entries" component_path=/ component_id=otelcol.receiver.loki.otlp err="sending queue is full"
ts=2024-11-27T02:30:05.980459907Z level=error msg="Exporting failed. Rejecting data." component_path=/ component_id=otelcol.exporter.otlp.otlp error="sending queue is full" rejected_items=1

Following is Helm values.yaml I am using:

cluster:
  name: my-cluster

destinations:
  - name: otlp
    type: otlp
    url: https://np-shared.com
    processors:
      attributes:
        actions:
          - key: my_cluster_label
            action: upsert
            value: my-cluster
    metrics:
      enabled: true
    logs:
      enabled: true
    tls:
      insecureSkipVerify: true

clusterMetrics:
  enabled: true
  kube-state-metrics:
    podAnnotations: {kubernetes.azure.com/set-kube-service-host-fqdn: "true"}

clusterEvents:
  enabled: true

podLogs:
  enabled: true

selfReporting:
  enabled: false

applicationObservability:
  enabled: true
  receivers:
    grpc:
      enabled: true

alloy-metrics:
  enabled: true
  controller:
    podAnnotations: {kubernetes.azure.com/set-kube-service-host-fqdn: "true"}

alloy-singleton:
  enabled: true
  controller:
    podAnnotations: {kubernetes.azure.com/set-kube-service-host-fqdn: "true"}

alloy-logs:
  enabled: true
  controller:
    podAnnotations: {kubernetes.azure.com/set-kube-service-host-fqdn: "true"}

alloy-receiver:
  enabled: true
  controller:
    podAnnotations: {kubernetes.azure.com/set-kube-service-host-fqdn: "true"}
  extraConfig: |-
      faro.receiver "integrations_app_agent_receiver" {
        server {
          listen_address           = "0.0.0.0"
          listen_port              = 8027
          cors_allowed_origins     = ["*"]
          max_allowed_payload_size = "10MiB"

          rate_limiting {
            rate = 100
          }
        }

        output {
          logs   = [otelcol.receiver.loki.otlp.receiver]
          traces = [otelcol.processor.transform.otlp.input]
        }
      }
  alloy:
    extraPorts:
      - name: otlp-grpc
        port: 4317
        targetPort: 4317
        protocol: TCP
      - name: faro
        port: 8027
        targetPort: 8027
        protocol: TCP
@petewall
Copy link
Collaborator

Check earlier in the logs for other error messages? It's possible that there are errors delivering the data to the destination and because of that, the batch is filling up.

@cxk314
Copy link
Author

cxk314 commented Nov 27, 2024 •
edited
Loading

Thanks @petewall. Checked earlier logs, not much there. Why would it generate this error?

ts=2024-11-27T03:20:21.254100357Z level=info "boringcrypto enabled"=false
ts=2024-11-27T03:20:21.253198969Z level=info source=/go/pkg/mod/github.com/!kim!machine!gun/automemlimit@v0.6.0/memlimit/memlimit.go:170 msg="memory is not limited, skipping: %v" package=github.com/KimMachineGun/automemlimit/memlimit !BADKEY="memory is not limited"
ts=2024-11-27T03:20:21.254147822Z level=info msg="no peer discovery configured: both join and discover peers are empty" service=cluster
ts=2024-11-27T03:20:21.254151741Z level=info msg="running usage stats reporter"
ts=2024-11-27T03:20:21.25415396Z level=info msg="starting complete graph evaluation" controller_path=/ controller_id="" trace_id=3a83a9e2764a608035626fa0facfc749
ts=2024-11-27T03:20:21.254160938Z level=info msg="finished node evaluation" controller_path=/ controller_id="" trace_id=3a83a9e2764a608035626fa0facfc749 node_id=labelstore duration=7.242µs
ts=2024-11-27T03:20:21.254166619Z level=info msg="finished node evaluation" controller_path=/ controller_id="" trace_id=3a83a9e2764a608035626fa0facfc749 node_id=logging duration=68.943µs
ts=2024-11-27T03:20:21.254173826Z level=info msg="finished node evaluation" controller_path=/ controller_id="" trace_id=3a83a9e2764a608035626fa0facfc749 node_id=otel duration=673ns
ts=2024-11-27T03:20:21.254191632Z level=info msg="finished node evaluation" controller_path=/ controller_id="" trace_id=3a83a9e2764a608035626fa0facfc749 node_id=livedebugging duration=7.342µs
ts=2024-11-27T03:20:21.254502891Z level=info msg="finished node evaluation" controller_path=/ controller_id="" trace_id=3a83a9e2764a608035626fa0facfc749 node_id=otelcol.exporter.otlp.otlp duration=292.112µs
ts=2024-11-27T03:20:21.257050248Z level=info msg="finished node evaluation" controller_path=/ controller_id="" trace_id=3a83a9e2764a608035626fa0facfc749 node_id=otelcol.processor.transform.otlp duration=2.523503ms
ts=2024-11-27T03:20:21.257125405Z level=info msg="finished node evaluation" controller_path=/ controller_id="" trace_id=3a83a9e2764a608035626fa0facfc749 node_id=otelcol.receiver.loki.otlp duration=47.329µs
ts=2024-11-27T03:20:21.257231913Z level=warn msg="this stdlib function is deprecated; please refer to the documentation for updated usage and alternatives" controller_path=/ controller_id=pod_logs.feature function=env
ts=2024-11-27T03:20:21.257266304Z level=info msg="starting complete graph evaluation" controller_path=/ controller_id=pod_logs.feature trace_id=25a809e0b76e8c2f16321b161cf6e261
ts=2024-11-27T03:20:21.25744042Z level=info msg="Using pod service account via in-cluster config" component_path=/pod_logs.feature component_id=discovery.kubernetes.pods
ts=2024-11-27T03:20:21.257698721Z level=info msg="finished node evaluation" controller_path=/ controller_id=pod_logs.feature trace_id=25a809e0b76e8c2f16321b161cf6e261 node_id=discovery.kubernetes.pods duration=421.794µs
ts=2024-11-27T03:20:21.257853137Z level=info msg="finished node evaluation" controller_path=/ controller_id=pod_logs.feature trace_id=25a809e0b76e8c2f16321b161cf6e261 node_id=discovery.relabel.filtered_pods duration=130.985µs
ts=2024-11-27T03:20:21.257891993Z level=info msg="finished node evaluation" controller_path=/ controller_id=pod_logs.feature trace_id=25a809e0b76e8c2f16321b161cf6e261 node_id=discovery.relabel.filtered_pods_with_paths duration=21.514µs
ts=2024-11-27T03:20:21.257916Z level=info msg="finished node evaluation" controller_path=/ controller_id=pod_logs.feature trace_id=25a809e0b76e8c2f16321b161cf6e261 node_id=local.file_match.pod_logs duration=12.849µs
ts=2024-11-27T03:20:21.257931772Z level=info msg="finished node evaluation" controller_path=/ controller_id=pod_logs.feature trace_id=25a809e0b76e8c2f16321b161cf6e261 node_id=argument.logs_destinations duration=9.055µs
ts=2024-11-27T03:20:21.258358254Z level=info msg="finished node evaluation" controller_path=/ controller_id=pod_logs.feature trace_id=25a809e0b76e8c2f16321b161cf6e261 node_id=loki.process.pod_logs duration=417.7µs
ts=2024-11-27T03:20:21.258544745Z level=info msg="finished node evaluation" controller_path=/ controller_id=pod_logs.feature trace_id=25a809e0b76e8c2f16321b161cf6e261 node_id=loki.source.file.pod_logs duration=167.119µs
ts=2024-11-27T03:20:21.258560477Z level=info msg="finished complete graph evaluation" controller_path=/ controller_id=pod_logs.feature trace_id=25a809e0b76e8c2f16321b161cf6e261 duration=1.37176ms
ts=2024-11-27T03:20:21.258568574Z level=info msg="finished node evaluation" controller_path=/ controller_id="" trace_id=3a83a9e2764a608035626fa0facfc749 node_id=pod_logs.feature duration=1.430606ms
ts=2024-11-27T03:20:21.258608318Z level=info msg="finished node evaluation" controller_path=/ controller_id="" trace_id=3a83a9e2764a608035626fa0facfc749 node_id=otelcol.receiver.prometheus.otlp duration=27.516µs
ts=2024-11-27T03:20:21.258622516Z level=info msg="finished node evaluation" controller_path=/ controller_id="" trace_id=3a83a9e2764a608035626fa0facfc749 node_id=declare.pod_logs duration=5.363µs
ts=2024-11-27T03:20:21.258661919Z level=info msg="finished node evaluation" controller_path=/ controller_id="" trace_id=3a83a9e2764a608035626fa0facfc749 node_id=remotecfg duration=34.021µs
ts=2024-11-27T03:20:21.258677068Z level=info msg="applying non-TLS config to HTTP server" service=http
ts=2024-11-27T03:20:21.258682005Z level=info msg="finished node evaluation" controller_path=/ controller_id="" trace_id=3a83a9e2764a608035626fa0facfc749 node_id=http duration=11.438µs
ts=2024-11-27T03:20:21.258696411Z level=info msg="finished node evaluation" controller_path=/ controller_id="" trace_id=3a83a9e2764a608035626fa0facfc749 node_id=cluster duration=4.527µs
ts=2024-11-27T03:20:21.258710157Z level=info msg="finished node evaluation" controller_path=/ controller_id="" trace_id=3a83a9e2764a608035626fa0facfc749 node_id=ui duration=4.826µs
ts=2024-11-27T03:20:21.258722721Z level=info msg="finished node evaluation" controller_path=/ controller_id="" trace_id=3a83a9e2764a608035626fa0facfc749 node_id=tracing duration=6.807µs
ts=2024-11-27T03:20:21.258730741Z level=info msg="finished complete graph evaluation" controller_path=/ controller_id="" trace_id=3a83a9e2764a608035626fa0facfc749 duration=4.74913ms
ts=2024-11-27T03:20:21.258785524Z level=info msg="scheduling loaded components and services"
ts=2024-11-27T03:20:21.2588924Z level=info msg="starting cluster node" service=cluster peers_count=0 peers="" advertise_addr=127.0.0.1:12345
ts=2024-11-27T03:20:21.258991037Z level=warn msg="this stdlib function is deprecated; please refer to the documentation for updated usage and alternatives" controller_path=/ controller_id=pod_logs.feature function=env
ts=2024-11-27T03:20:21.259023661Z level=info msg="starting complete graph evaluation" controller_path=/ controller_id=pod_logs.feature trace_id=5435517bace20c5e0b5ff7718766bc35
ts=2024-11-27T03:20:21.259089222Z level=info msg="peers changed" service=cluster peers_count=1 peers=ck-alloy-logs-dwgjl
ts=2024-11-27T03:20:21.259102196Z level=info msg="finished node evaluation" controller_path=/ controller_id=pod_logs.feature trace_id=5435517bace20c5e0b5ff7718766bc35 node_id=discovery.kubernetes.pods duration=64.799µs
ts=2024-11-27T03:20:21.259283956Z level=info msg="finished node evaluation" controller_path=/ controller_id=pod_logs.feature trace_id=5435517bace20c5e0b5ff7718766bc35 node_id=discovery.relabel.filtered_pods duration=168.306µs
ts=2024-11-27T03:20:21.259337218Z level=info msg="finished node evaluation" controller_path=/ controller_id=pod_logs.feature trace_id=5435517bace20c5e0b5ff7718766bc35 node_id=discovery.relabel.filtered_pods_with_paths duration=35.155µs
ts=2024-11-27T03:20:21.259449595Z level=info msg="finished node evaluation" controller_path=/ controller_id=pod_logs.feature trace_id=5435517bace20c5e0b5ff7718766bc35 node_id=local.file_match.pod_logs duration=90.886µs
ts=2024-11-27T03:20:21.259488658Z level=info msg="finished node evaluation" controller_path=/ controller_id=pod_logs.feature trace_id=5435517bace20c5e0b5ff7718766bc35 node_id=argument.logs_destinations duration=12.709µs
ts=2024-11-27T03:20:21.259629316Z level=info msg="now listening for http traffic" service=http addr=0.0.0.0:12345
ts=2024-11-27T03:20:21.260027396Z level=info msg="finished node evaluation" controller_path=/ controller_id=pod_logs.feature trace_id=5435517bace20c5e0b5ff7718766bc35 node_id=loki.process.pod_logs duration=524.003µs
ts=2024-11-27T03:20:21.260088407Z level=info msg="finished node evaluation" controller_path=/ controller_id=pod_logs.feature trace_id=5435517bace20c5e0b5ff7718766bc35 node_id=loki.source.file.pod_logs duration=29.402µs
ts=2024-11-27T03:20:21.260107246Z level=info msg="finished complete graph evaluation" controller_path=/ controller_id=pod_logs.feature trace_id=5435517bace20c5e0b5ff7718766bc35 duration=1.165119ms
ts=2024-11-27T03:20:21.260204723Z level=info msg="scheduling loaded components and services"
ts=2024-11-27T03:20:31.265079201Z level=info msg="tail routine: started" component_path=/pod_logs.feature component_id=loki.source.file.pod_logs component=tailer path=/var/log/pods/kube-system_cloud-node-manager-2tmqc_decf3b37-f24d-4874-9da4-70e2c1e44902/cloud-node-manager/0.log
ts=2024-11-27T03:20:31.265111487Z level=info msg="Seeked /var/log/pods/kube-system_cloud-node-manager-2tmqc_decf3b37-f24d-4874-9da4-70e2c1e44902/cloud-node-manager/0.log - &{Offset:0 Whence:0}" component_path=/pod_logs.feature component_id=loki.source.file.pod_logs
ts=2024-11-27T03:20:31.265135158Z level=info msg="tail routine: started" component_path=/pod_logs.feature component_id=loki.source.file.pod_logs component=tailer path=/var/log/pods/kube-system_konnectivity-agent-75f6bb7654-5x2p5_0b5a8fd5-c1c2-4d59-b29c-13879f01d65c/konnectivity-agent/0.log
ts=2024-11-27T03:20:31.265155515Z level=info msg="Seeked /var/log/pods/wiz_wiz-helm-wiz-sensor-2vp6r_bd1802f6-a53a-484d-8061-88365d7d2179/wiz-sensor/0.log - &{Offset:0 Whence:0}" component_path=/pod_logs.feature component_id=loki.source.file.pod_logs
ts=2024-11-27T03:20:31.265147409Z level=info msg="tail routine: started" component_path=/pod_logs.feature component_id=loki.source.file.pod_logs component=tailer path=/var/log/pods/wiz_wiz-helm-wiz-sensor-2vp6r_bd1802f6-a53a-484d-8061-88365d7d2179/wiz-sensor/0.log
ts=2024-11-27T03:20:31.265122178Z level=info msg="Seeked /var/log/pods/kube-system_konnectivity-agent-75f6bb7654-5x2p5_0b5a8fd5-c1c2-4d59-b29c-13879f01d65c/konnectivity-agent/0.log - &{Offset:0 Whence:0}" component_path=/pod_logs.feature component_id=loki.source.file.pod_logs
ts=2024-11-27T03:20:31.265227969Z level=info msg="Seeked /var/log/pods/dapr-system_dapr-operator-559ff985fd-vbtb8_a59e56c7-23fa-4b16-90a3-53f204badf6a/dapr-operator/0.log - &{Offset:0 Whence:0}" component_path=/pod_logs.feature component_id=loki.source.file.pod_logs
ts=2024-11-27T03:20:31.265252563Z level=info msg="tail routine: started" component_path=/pod_logs.feature component_id=loki.source.file.pod_logs component=tailer path=/var/log/pods/dapr-system_dapr-operator-559ff985fd-vbtb8_a59e56c7-23fa-4b16-90a3-53f204badf6a/dapr-operator/0.log
ts=2024-11-27T03:20:31.265307738Z level=info msg="Seeked /var/log/pods/dev-upt-backstage_upt-mfe-todos-deployment-5b8b4b647f-7jgns_0c346040-4bf4-4c9b-852b-13567bf3af6a/upt-mfe-todos/0.log - &{Offset:0 Whence:0}" component_path=/pod_logs.feature component_id=loki.source.file.pod_logs
ts=2024-11-27T03:20:31.265329881Z level=info msg="tail routine: started" component_path=/pod_logs.feature component_id=loki.source.file.pod_logs component=tailer path=/var/log/pods/dev-upt-backstage_upt-mfe-todos-deployment-5b8b4b647f-7jgns_0c346040-4bf4-4c9b-852b-13567bf3af6a/upt-mfe-todos/0.log
ts=2024-11-27T03:20:31.265346463Z level=info msg="Seeked /var/log/pods/dev-upt-backstage_upt-mfe-todos-deployment-5b8b4b647f-7jgns_0c346040-4bf4-4c9b-852b-13567bf3af6a/vault-agent/0.log - &{Offset:0 Whence:0}" component_path=/pod_logs.feature component_id=loki.source.file.pod_logs
ts=2024-11-27T03:20:31.265377291Z level=info msg="tail routine: started" component_path=/pod_logs.feature component_id=loki.source.file.pod_logs component=tailer path=/var/log/pods/dev-upt-backstage_upt-mfe-todos-deployment-5b8b4b647f-7jgns_0c346040-4bf4-4c9b-852b-13567bf3af6a/vault-agent/0.log
ts=2024-11-27T03:20:31.265434994Z level=info msg="tail routine: started" component_path=/pod_logs.feature component_id=loki.source.file.pod_logs component=tailer path=/var/log/pods/dev-upt-backstage_upt-mfe-todos-deployment-5b8b4b647f-7jgns_0c346040-4bf4-4c9b-852b-13567bf3af6a/vault-agent-init/0.log
ts=2024-11-27T03:20:31.265455961Z level=info msg="Seeked /var/log/pods/dev-upt-backstage_upt-mfe-todos-deployment-5b8b4b647f-7jgns_0c346040-4bf4-4c9b-852b-13567bf3af6a/vault-agent-init/0.log - &{Offset:0 Whence:0}" component_path=/pod_logs.feature component_id=loki.source.file.pod_logs
ts=2024-11-27T03:20:31.265463539Z level=info msg="Seeked /var/log/pods/dev-upt-backstage_upt-todo-bfe-net-69c9bff85c-7xjrn_6ee57591-b838-4173-b363-c7cd0a4854ed/upt-todo-bfe-net/0.log - &{Offset:0 Whence:0}" component_path=/pod_logs.feature component_id=loki.source.file.pod_logs
ts=2024-11-27T03:20:31.2654723Z level=info msg="tail routine: started" component_path=/pod_logs.feature component_id=loki.source.file.pod_logs component=tailer path=/var/log/pods/dev-upt-backstage_upt-todo-bfe-net-69c9bff85c-7xjrn_6ee57591-b838-4173-b363-c7cd0a4854ed/upt-todo-bfe-net/0.log
ts=2024-11-27T03:20:31.265557361Z level=info msg="Seeked /var/log/pods/kube-system_kube-proxy-2qw4m_a2cc56a3-e784-4629-bd1a-314045c4654e/kube-proxy/0.log - &{Offset:0 Whence:0}" component_path=/pod_logs.feature component_id=loki.source.file.pod_logs
ts=2024-11-27T03:20:31.265559603Z level=info msg="Seeked /var/log/pods/kube-system_azure-ip-masq-agent-t4bgt_7c3ade12-7ba9-45d4-b772-ad8e8acc676a/azure-ip-masq-agent/0.log - &{Offset:0 Whence:0}" component_path=/pod_logs.feature component_id=loki.source.file.pod_logs
ts=2024-11-27T03:20:31.265574682Z level=info msg="tail routine: started" component_path=/pod_logs.feature component_id=loki.source.file.pod_logs component=tailer path=/var/log/pods/kube-system_kube-proxy-2qw4m_a2cc56a3-e784-4629-bd1a-314045c4654e/kube-proxy/0.log
ts=2024-11-27T03:20:31.265588835Z level=info msg="tail routine: started" component_path=/pod_logs.feature component_id=loki.source.file.pod_logs component=tailer path=/var/log/pods/kube-system_azure-ip-masq-agent-t4bgt_7c3ade12-7ba9-45d4-b772-ad8e8acc676a/azure-ip-masq-agent/0.log
ts=2024-11-27T03:20:31.265676136Z level=info msg="tail routine: started" component_path=/pod_logs.feature component_id=loki.source.file.pod_logs component=tailer path=/var/log/pods/kube-system_metrics-server-677fd8c788-pj64h_0caf1483-3479-4cec-942a-85c44a85bac6/metrics-server-vpa/0.log
ts=2024-11-27T03:20:31.265680354Z level=info msg="Seeked /var/log/pods/kube-system_kube-proxy-2qw4m_a2cc56a3-e784-4629-bd1a-314045c4654e/kube-proxy-bootstrap/0.log - &{Offset:0 Whence:0}" component_path=/pod_logs.feature component_id=loki.source.file.pod_logs
ts=2024-11-27T03:20:31.265724376Z level=info msg="tail routine: started" component_path=/pod_logs.feature component_id=loki.source.file.pod_logs component=tailer path=/var/log/pods/kube-system_metrics-server-677fd8c788-pj64h_0caf1483-3479-4cec-942a-85c44a85bac6/metrics-server/0.log
ts=2024-11-27T03:20:31.26573942Z level=info msg="Seeked /var/log/pods/kube-system_metrics-server-677fd8c788-pj64h_0caf1483-3479-4cec-942a-85c44a85bac6/metrics-server-vpa/0.log - &{Offset:0 Whence:0}" component_path=/pod_logs.feature component_id=loki.source.file.pod_logs
ts=2024-11-27T03:20:31.265749792Z level=info msg="Seeked /var/log/pods/kube-system_metrics-server-677fd8c788-pj64h_0caf1483-3479-4cec-942a-85c44a85bac6/metrics-server/0.log - &{Offset:0 Whence:0}" component_path=/pod_logs.feature component_id=loki.source.file.pod_logs
ts=2024-11-27T03:20:31.265698591Z level=info msg="tail routine: started" component_path=/pod_logs.feature component_id=loki.source.file.pod_logs component=tailer path=/var/log/pods/kube-system_kube-proxy-2qw4m_a2cc56a3-e784-4629-bd1a-314045c4654e/kube-proxy-bootstrap/0.log
ts=2024-11-27T03:20:31.265831796Z level=info msg="Seeked /var/log/pods/kube-system_nginx-ingress-ingress-nginx-controller-7bfbfd899d-h2wmn_38a0edf2-9fa3-4679-b224-5f9979570f7a/controller/0.log - &{Offset:0 Whence:0}" component_path=/pod_logs.feature component_id=loki.source.file.pod_logs
ts=2024-11-27T03:20:31.265916455Z level=info msg="Seeked /var/log/pods/vault_vault-agent-injector-agent-injector-846b7d74f8-6wrd8_96d40a90-d985-4c4d-b869-cf8f717be49d/sidecar-injector/0.log - &{Offset:0 Whence:0}" component_path=/pod_logs.feature component_id=loki.source.file.pod_logs
ts=2024-11-27T03:20:31.265916568Z level=info msg="tail routine: started" component_path=/pod_logs.feature component_id=loki.source.file.pod_logs component=tailer path=/var/log/pods/vault_vault-agent-injector-agent-injector-846b7d74f8-6wrd8_96d40a90-d985-4c4d-b869-cf8f717be49d/sidecar-injector/0.log
ts=2024-11-27T03:20:31.265981776Z level=info msg="Seeked /var/log/pods/dapr-system_dapr-placement-server-0_77a98358-33e3-4760-82fc-8ad073fda6af/dapr-placement-server/0.log - &{Offset:0 Whence:0}" component_path=/pod_logs.feature component_id=loki.source.file.pod_logs
ts=2024-11-27T03:20:31.265988888Z level=info msg="tail routine: started" component_path=/pod_logs.feature component_id=loki.source.file.pod_logs component=tailer path=/var/log/pods/dapr-system_dapr-placement-server-0_77a98358-33e3-4760-82fc-8ad073fda6af/dapr-placement-server/0.log
ts=2024-11-27T03:20:31.266050603Z level=info msg="tail routine: started" component_path=/pod_logs.feature component_id=loki.source.file.pod_logs component=tailer path=/var/log/pods/dapr-system_dapr-sidecar-injector-65fd7d94c5-vlb7z_ae601c13-2f66-451f-9e9b-8031279dc989/dapr-sidecar-injector/1.log
ts=2024-11-27T03:20:31.26608141Z level=info msg="Seeked /var/log/pods/dapr-system_dapr-sidecar-injector-65fd7d94c5-vlb7z_ae601c13-2f66-451f-9e9b-8031279dc989/dapr-sidecar-injector/1.log - &{Offset:0 Whence:0}" component_path=/pod_logs.feature component_id=loki.source.file.pod_logs
ts=2024-11-27T03:20:31.266096493Z level=info msg="Seeked /var/log/pods/dapr-system_dapr-sidecar-injector-65fd7d94c5-vlb7z_ae601c13-2f66-451f-9e9b-8031279dc989/dapr-sidecar-injector/2.log - &{Offset:0 Whence:0}" component_path=/pod_logs.feature component_id=loki.source.file.pod_logs
ts=2024-11-27T03:20:31.266117045Z level=info msg="tail routine: started" component_path=/pod_logs.feature component_id=loki.source.file.pod_logs component=tailer path=/var/log/pods/dapr-system_dapr-sidecar-injector-65fd7d94c5-vlb7z_ae601c13-2f66-451f-9e9b-8031279dc989/dapr-sidecar-injector/2.log
ts=2024-11-27T03:20:31.265995358Z level=info msg="tail routine: started" component_path=/pod_logs.feature component_id=loki.source.file.pod_logs component=tailer path=/var/log/pods/kube-system_nginx-ingress-ingress-nginx-controller-7bfbfd899d-h2wmn_38a0edf2-9fa3-4679-b224-5f9979570f7a/controller/0.log
ts=2024-11-27T03:20:31.266233318Z level=info msg="Seeked /var/log/pods/wiz_wiz-helm-wiz-admission-controller-6cf7cb56f8-k5x4b_a109d818-3c30-47af-bacd-71092ced218e/wiz-admission-controller/0.log - &{Offset:0 Whence:0}" component_path=/pod_logs.feature component_id=loki.source.file.pod_logs
ts=2024-11-27T03:20:31.266213722Z level=info msg="tail routine: started" component_path=/pod_logs.feature component_id=loki.source.file.pod_logs component=tailer path=/var/log/pods/wiz_wiz-helm-wiz-admission-controller-6cf7cb56f8-k5x4b_a109d818-3c30-47af-bacd-71092ced218e/wiz-admission-controller/0.log
ts=2024-11-27T03:20:31.266300143Z level=info msg="Seeked /var/log/pods/ck_ck-alloy-logs-dwgjl_869c8a3b-6b5b-4012-a7ca-4609ff63d145/alloy/0.log - &{Offset:0 Whence:0}" component_path=/pod_logs.feature component_id=loki.source.file.pod_logs
ts=2024-11-27T03:20:31.26635855Z level=info msg="tail routine: started" component_path=/pod_logs.feature component_id=loki.source.file.pod_logs component=tailer path=/var/log/pods/ck_ck-alloy-logs-dwgjl_869c8a3b-6b5b-4012-a7ca-4609ff63d145/config-reloader/0.log
ts=2024-11-27T03:20:31.266417841Z level=info msg="Seeked /var/log/pods/kube-system_coredns-597bb9d4db-gn7fn_239d9c70-ab90-4125-be86-35f19a419fc4/coredns/0.log - &{Offset:0 Whence:0}" component_path=/pod_logs.feature component_id=loki.source.file.pod_logs
ts=2024-11-27T03:20:31.266479018Z level=info msg="tail routine: started" component_path=/pod_logs.feature component_id=loki.source.file.pod_logs component=tailer path=/var/log/pods/kube-system_csi-azuredisk-node-sm6nv_88e5b1fa-b2b0-4463-af82-f3ec38c29555/liveness-probe/0.log
ts=2024-11-27T03:20:31.266522079Z level=info msg="Seeked /var/log/pods/kube-system_csi-azuredisk-node-sm6nv_88e5b1fa-b2b0-4463-af82-f3ec38c29555/node-driver-registrar/0.log - &{Offset:0 Whence:0}" component_path=/pod_logs.feature component_id=loki.source.file.pod_logs
ts=2024-11-27T03:20:31.266580263Z level=info msg="tail routine: started" component_path=/pod_logs.feature component_id=loki.source.file.pod_logs component=tailer path=/var/log/pods/kube-system_csi-azuredisk-node-sm6nv_88e5b1fa-b2b0-4463-af82-f3ec38c29555/azuredisk/0.log
ts=2024-11-27T03:20:31.266712135Z level=info msg="Seeked /var/log/pods/kube-system_csi-azurefile-node-dxx7j_935aa036-28a6-4c43-8e8c-78f32d7e86a5/liveness-probe/0.log - &{Offset:0 Whence:0}" component_path=/pod_logs.feature component_id=loki.source.file.pod_logs
ts=2024-11-27T03:20:31.266277604Z level=info msg="tail routine: started" component_path=/pod_logs.feature component_id=loki.source.file.pod_logs component=tailer path=/var/log/pods/ck_ck-alloy-logs-dwgjl_869c8a3b-6b5b-4012-a7ca-4609ff63d145/alloy/0.log
ts=2024-11-27T03:20:31.266853051Z level=info msg="Seeked /var/log/pods/ck_ck-alloy-logs-dwgjl_869c8a3b-6b5b-4012-a7ca-4609ff63d145/config-reloader/0.log - &{Offset:0 Whence:0}" component_path=/pod_logs.feature component_id=loki.source.file.pod_logs
ts=2024-11-27T03:20:31.266448101Z level=info msg="tail routine: started" component_path=/pod_logs.feature component_id=loki.source.file.pod_logs component=tailer path=/var/log/pods/kube-system_coredns-597bb9d4db-gn7fn_239d9c70-ab90-4125-be86-35f19a419fc4/coredns/0.log
ts=2024-11-27T03:20:31.266917055Z level=info msg="Seeked /var/log/pods/kube-system_csi-azuredisk-node-sm6nv_88e5b1fa-b2b0-4463-af82-f3ec38c29555/liveness-probe/0.log - &{Offset:0 Whence:0}" component_path=/pod_logs.feature component_id=loki.source.file.pod_logs
ts=2024-11-27T03:20:31.266504494Z level=info msg="tail routine: started" component_path=/pod_logs.feature component_id=loki.source.file.pod_logs component=tailer path=/var/log/pods/kube-system_csi-azuredisk-node-sm6nv_88e5b1fa-b2b0-4463-af82-f3ec38c29555/node-driver-registrar/0.log
ts=2024-11-27T03:20:31.266936603Z level=info msg="Seeked /var/log/pods/kube-system_csi-azuredisk-node-sm6nv_88e5b1fa-b2b0-4463-af82-f3ec38c29555/azuredisk/0.log - &{Offset:0 Whence:0}" component_path=/pod_logs.feature component_id=loki.source.file.pod_logs
ts=2024-11-27T03:20:31.26693311Z level=info msg="tail routine: started" component_path=/pod_logs.feature component_id=loki.source.file.pod_logs component=tailer path=/var/log/pods/ck_ck-alloy-receiver-6mgrd_ad133fc4-f0c4-4931-8cb1-fb8bf547031c/alloy/0.log
ts=2024-11-27T03:20:31.266975239Z level=info msg="Seeked /var/log/pods/ck_ck-node-exporter-prhnq_f3b4cf80-5972-47c3-82ef-05e3dd5823da/node-exporter/0.log - &{Offset:0 Whence:0}" component_path=/pod_logs.feature component_id=loki.source.file.pod_logs
ts=2024-11-27T03:20:31.266997963Z level=info msg="tail routine: started" component_path=/pod_logs.feature component_id=loki.source.file.pod_logs component=tailer path=/var/log/pods/ck_ck-alloy-receiver-6mgrd_ad133fc4-f0c4-4931-8cb1-fb8bf547031c/config-reloader/0.log
ts=2024-11-27T03:20:31.26701104Z level=info msg="Seeked /var/log/pods/ck_ck-alloy-receiver-6mgrd_ad133fc4-f0c4-4931-8cb1-fb8bf547031c/config-reloader/0.log - &{Offset:0 Whence:0}" component_path=/pod_logs.feature component_id=loki.source.file.pod_logs
ts=2024-11-27T03:20:31.267012431Z level=info msg="tail routine: started" component_path=/pod_logs.feature component_id=loki.source.file.pod_logs component=tailer path=/var/log/pods/ck_ck-node-exporter-prhnq_f3b4cf80-5972-47c3-82ef-05e3dd5823da/node-exporter/0.log
ts=2024-11-27T03:20:31.267019687Z level=info msg="Seeked /var/log/pods/kube-system_csi-azurefile-node-dxx7j_935aa036-28a6-4c43-8e8c-78f32d7e86a5/azurefile/0.log - &{Offset:0 Whence:0}" component_path=/pod_logs.feature component_id=loki.source.file.pod_logs
ts=2024-11-27T03:20:31.266968211Z level=info msg="tail routine: started" component_path=/pod_logs.feature component_id=loki.source.file.pod_logs component=tailer path=/var/log/pods/kube-system_zscaler-ca-nm2g6_872c3306-8c25-4152-8516-4acfc864b59c/zscaler-ca/0.log
ts=2024-11-27T03:20:31.266619977Z level=info msg="tail routine: started" component_path=/pod_logs.feature component_id=loki.source.file.pod_logs component=tailer path=/var/log/pods/kube-system_csi-azurefile-node-dxx7j_935aa036-28a6-4c43-8e8c-78f32d7e86a5/liveness-probe/0.log
ts=2024-11-27T03:20:31.267040629Z level=info msg="tail routine: started" component_path=/pod_logs.feature component_id=loki.source.file.pod_logs component=tailer path=/var/log/pods/dapr-system_dapr-sentry-7f6966fd59-6dpmx_3c3ab88e-3365-45af-958f-eee3ad421025/dapr-sentry/0.log
ts=2024-11-27T03:20:31.267045504Z level=info msg="Seeked /var/log/pods/ck_ck-alloy-receiver-6mgrd_ad133fc4-f0c4-4931-8cb1-fb8bf547031c/alloy/0.log - &{Offset:0 Whence:0}" component_path=/pod_logs.feature component_id=loki.source.file.pod_logs
ts=2024-11-27T03:20:31.267054178Z level=info msg="Seeked /var/log/pods/dapr-system_dapr-sentry-7f6966fd59-6dpmx_3c3ab88e-3365-45af-958f-eee3ad421025/dapr-sentry/0.log - &{Offset:0 Whence:0}" component_path=/pod_logs.feature component_id=loki.source.file.pod_logs
ts=2024-11-27T03:20:31.267077667Z level=info msg="Seeked /var/log/pods/kube-system_zscaler-ca-nm2g6_872c3306-8c25-4152-8516-4acfc864b59c/zscaler-ca/0.log - &{Offset:0 Whence:0}" component_path=/pod_logs.feature component_id=loki.source.file.pod_logs
ts=2024-11-27T03:20:31.267162651Z level=info msg="tail routine: started" component_path=/pod_logs.feature component_id=loki.source.file.pod_logs component=tailer path=/var/log/pods/kube-system_csi-azurefile-node-dxx7j_935aa036-28a6-4c43-8e8c-78f32d7e86a5/node-driver-registrar/0.log
ts=2024-11-27T03:20:31.267161907Z level=info msg="tail routine: started" component_path=/pod_logs.feature component_id=loki.source.file.pod_logs component=tailer path=/var/log/pods/kube-system_csi-azurefile-node-dxx7j_935aa036-28a6-4c43-8e8c-78f32d7e86a5/azurefile/0.log
ts=2024-11-27T03:20:31.26732654Z level=info msg="Seeked /var/log/pods/kube-system_csi-azurefile-node-dxx7j_935aa036-28a6-4c43-8e8c-78f32d7e86a5/node-driver-registrar/0.log - &{Offset:0 Whence:0}" component_path=/pod_logs.feature component_id=loki.source.file.pod_logs
ts=2024-11-27T03:20:31.28990363Z level=error msg="Exporting failed. Rejecting data." component_path=/ component_id=otelcol.exporter.otlp.otlp error="sending queue is full" rejected_items=1
ts=2024-11-27T03:20:31.289931335Z level=error msg="failed to consume log entries" component_path=/ component_id=otelcol.receiver.loki.otlp err="sending queue is full"
ts=2024-11-27T03:20:31.289960789Z level=error msg="Exporting failed. Rejecting data." component_path=/ component_id=otelcol.exporter.otlp.otlp error="sending queue is full" rejected_items=1
ts=2024-11-27T03:20:31.28997012Z level=error msg="failed to consume log entries" component_path=/ component_id=otelcol.receiver.loki.otlp err="sending queue is full"
ts=2024-11-27T03:20:31.289993192Z level=error msg="Exporting failed. Rejecting data." component_path=/ component_id=otelcol.exporter.otlp.otlp error="sending queue is full" rejected_items=1

@cxk314
Copy link
Author

cxk314 commented Nov 27, 2024

The previous error appears fixed in 2.0.0-rc.6, probably because buffering was released.

However, there is different error now:

level=error msg="Exporting failed. Dropping data." component_path=/ component_id=otelcol.exporter.otlp.otlp error="not retryable error: Permanent error: rpc error: code = ResourceExhausted desc = grpc: received message after decompression larger than max (5440945 vs. 4194304)" dropped_items=8192
ts=2024-11-27T23:36:31.327950063Z level=error msg="Exporting failed. Dropping data." component_path=/ component_id=otelcol.exporter.otlp.otlp error="not retryable error: Permanent error: rpc error: code = ResourceExhausted desc = grpc: received message after decompression larger than max (5580877 vs. 4194304)" dropped_items=8192
ts=2024-11-27T23:36:31.556445725Z level=error msg="Exporting failed. Dropping data." component_path=/ component_id=otelcol.exporter.otlp.otlp error="not retryable error: Permanent error: rpc error: code = ResourceExhausted desc = grpc: received message after decompression larger than max (5757119 vs. 4194304)" dropped_items=8192
ts=2024-11-27T23:36:31.772304661Z level=error msg="Exporting failed. Dropping data." component_path=/ component_id=otelcol.exporter.otlp.otlp error="not retryable error: Permanent error: rpc error: code = ResourceExhausted desc = grpc: received message after decompression larger than max (5807642 vs. 4194304)" dropped_items=8192

@petewall
Copy link
Collaborator

try setting the max size field for the otlp destination batch:

destinations:
- name ...
  type: otlp
  processors:
    batch:
      maxSize: <something larger than 0>

https://grafana.com/docs/alloy/latest/reference/components/otelcol/otelcol.processor.batch/#arguments

@cxk314
Copy link
Author

cxk314 commented Nov 29, 2024

Fixed it by doing following, both size, and maxSize are necessary.

destinations:
  - name: otlp
    type: otlp
    url: https://np-grpc.np-shared.com
    processors:
      batch:
        size: 4000
        maxSize: 4000

Getting the below error now:

ts=2024-11-29T14:08:57.666380305Z level=info msg="Exporting failed. Will retry the request after interval." component_path=/ component_id=otelcol.exporter.otlp.otlp error="rpc error: code = Unavailable desc = unexpected HTTP status code received from server: 503 (Service Unavailable); transport: received unexpected content-type \"text/html\"" interval=4.076995382s

@petewall
Copy link
Collaborator

Getting the below error now:

I see you have another issue opened for this #962. I'm going to close this one and we can chat about the other issue over there.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@petewall @cxk314