We read every piece of feedback, and take your input very seriously.
To see all available qualifiers, see our documentation.
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Pods are stuck in CrashLoopBackOff:
kubectl logs -n monitoring pod/k8s-monitoring-alloy-logs-b4mxk ts=2024-12-29T14:20:50.624177388Z level=info "boringcrypto enabled"=false ts=2024-12-29T14:20:50.623108985Z level=info source=/go/pkg/mod/github.com/!kim!machine!gun/automemlimit@v0.6.0/memlimit/memlimit.go:176 msg="GOMEMLIMIT is updated" package=github.com/KimMachineGun/automemlimit/memlimit GOMEMLIMIT=8263224115 ts=2024-12-29T14:20:50.624286349Z level=info msg="no peer discovery configured: both join and discover peers are empty" service=cluster ts=2024-12-29T14:20:50.624291229Z level=info msg="running usage stats reporter" ts=2024-12-29T14:20:50.624294189Z level=info msg="starting complete graph evaluation" controller_path=/ controller_id="" trace_id=c64a29d79689717a4f787fda6cccc0e9 ts=2024-12-29T14:20:50.624304629Z level=info msg="finished node evaluation" controller_path=/ controller_id="" trace_id=c64a29d79689717a4f787fda6cccc0e9 node_id=logging duration=131.401µs ts=2024-12-29T14:20:50.624314829Z level=info msg="finished node evaluation" controller_path=/ controller_id="" trace_id=c64a29d79689717a4f787fda6cccc0e9 node_id=declare.pod_logs duration=1.52µs ts=2024-12-29T14:20:50.624370189Z level=info msg="finished node evaluation" controller_path=/ controller_id="" trace_id=c64a29d79689717a4f787fda6cccc0e9 node_id=remotecfg duration=46.24µs ts=2024-12-29T14:20:50.624384509Z level=info msg="applying non-TLS config to HTTP server" service=http ts=2024-12-29T14:20:50.624389549Z level=info msg="finished node evaluation" controller_path=/ controller_id="" trace_id=c64a29d79689717a4f787fda6cccc0e9 node_id=http duration=9.64µs ts=2024-12-29T14:20:50.624397549Z level=info msg="finished node evaluation" controller_path=/ controller_id="" trace_id=c64a29d79689717a4f787fda6cccc0e9 node_id=cluster duration=600ns ts=2024-12-29T14:20:50.624406789Z level=info msg="finished node evaluation" controller_path=/ controller_id="" trace_id=c64a29d79689717a4f787fda6cccc0e9 node_id=otel duration=680ns ts=2024-12-29T14:20:50.624418069Z level=info msg="finished node evaluation" controller_path=/ controller_id="" trace_id=c64a29d79689717a4f787fda6cccc0e9 node_id=livedebugging duration=4.2µs ts=2024-12-29T14:20:50.624426709Z level=info msg="finished node evaluation" controller_path=/ controller_id="" trace_id=c64a29d79689717a4f787fda6cccc0e9 node_id=ui duration=600ns ts=2024-12-29T14:20:50.624992471Z level=info msg="finished node evaluation" controller_path=/ controller_id="" trace_id=c64a29d79689717a4f787fda6cccc0e9 node_id=loki.write.loki duration=556.722µs ts=2024-12-29T14:20:50.625146591Z level=info msg="starting complete graph evaluation" controller_path=/ controller_id=node_logs.feature trace_id=09469a8f75857b1e951268b1cc467a70 ts=2024-12-29T14:20:50.625459112Z level=info msg="finished node evaluation" controller_path=/ controller_id=node_logs.feature trace_id=09469a8f75857b1e951268b1cc467a70 node_id=loki.relabel.journal duration=296.001µs ts=2024-12-29T14:20:50.625517592Z level=info msg="finished node evaluation" controller_path=/ controller_id=node_logs.feature trace_id=09469a8f75857b1e951268b1cc467a70 node_id=argument.logs_destinations duration=32.4µs ts=2024-12-29T14:20:50.626026273Z level=info msg="finished node evaluation" controller_path=/ controller_id=node_logs.feature trace_id=09469a8f75857b1e951268b1cc467a70 node_id=loki.process.journal_logs duration=492.881µs ts=2024-12-29T14:20:50.627363197Z level=error msg="failed to evaluate config" controller_path=/ controller_id=node_logs.feature trace_id=09469a8f75857b1e951268b1cc467a70 node=loki.source.journal.worker err="building component: creating journal reader: failed to open journal in directory \"/var/log/journal\": no such file or directory" ts=2024-12-29T14:20:50.627411037Z level=info msg="finished node evaluation" controller_path=/ controller_id=node_logs.feature trace_id=09469a8f75857b1e951268b1cc467a70 node_id=loki.source.journal.worker duration=1.333723ms ts=2024-12-29T14:20:50.627425157Z level=info msg="finished complete graph evaluation" controller_path=/ controller_id=node_logs.feature trace_id=09469a8f75857b1e951268b1cc467a70 duration=2.335846ms ts=2024-12-29T14:20:50.627445677Z level=error msg="failed to evaluate config" controller_path=/ controller_id="" trace_id=c64a29d79689717a4f787fda6cccc0e9 node=node_logs.feature err="updating custom component: /etc/alloy/config.alloy:72:3: Failed to build component: building component: creating journal reader: failed to open journal in directory \"/var/log/journal\": no such file or directory" ts=2024-12-29T14:20:50.627459637Z level=info msg="finished node evaluation" controller_path=/ controller_id="" trace_id=c64a29d79689717a4f787fda6cccc0e9 node_id=node_logs.feature duration=2.445886ms ts=2024-12-29T14:20:50.627582998Z level=info msg="finished node evaluation" controller_path=/ controller_id="" trace_id=c64a29d79689717a4f787fda6cccc0e9 node_id=otelcol.exporter.loki.loki duration=111.601µs ts=2024-12-29T14:20:50.627747798Z level=info msg="starting complete graph evaluation" controller_path=/ controller_id=pod_logs.feature trace_id=19121e540c1f0a565467f45cd5bef006 ts=2024-12-29T14:20:50.627777918Z level=info msg="finished node evaluation" controller_path=/ controller_id=pod_logs.feature trace_id=19121e540c1f0a565467f45cd5bef006 node_id=argument.logs_destinations duration=13.64µs ts=2024-12-29T14:20:50.6284452Z level=info msg="finished node evaluation" controller_path=/ controller_id=pod_logs.feature trace_id=19121e540c1f0a565467f45cd5bef006 node_id=loki.process.pod_logs duration=653.482µs ts=2024-12-29T14:20:50.628703721Z level=info msg="Using pod service account via in-cluster config" component_path=/pod_logs.feature component_id=discovery.kubernetes.pods ts=2024-12-29T14:20:50.628913361Z level=info msg="finished node evaluation" controller_path=/ controller_id=pod_logs.feature trace_id=19121e540c1f0a565467f45cd5bef006 node_id=discovery.kubernetes.pods duration=441.721µs ts=2024-12-29T14:20:50.629295562Z level=info msg="finished node evaluation" controller_path=/ controller_id=pod_logs.feature trace_id=19121e540c1f0a565467f45cd5bef006 node_id=discovery.relabel.filtered_pods duration=359.521µs ts=2024-12-29T14:20:50.629377203Z level=info msg="finished node evaluation" controller_path=/ controller_id=pod_logs.feature trace_id=19121e540c1f0a565467f45cd5bef006 node_id=discovery.relabel.filtered_pods_with_paths duration=59.481µs ts=2024-12-29T14:20:50.629419443Z level=info msg="finished node evaluation" controller_path=/ controller_id=pod_logs.feature trace_id=19121e540c1f0a565467f45cd5bef006 node_id=local.file_match.pod_logs duration=23.56µs ts=2024-12-29T14:20:50.629698044Z level=info msg="finished node evaluation" controller_path=/ controller_id=pod_logs.feature trace_id=19121e540c1f0a565467f45cd5bef006 node_id=loki.source.file.pod_logs duration=262.681µs ts=2024-12-29T14:20:50.629718644Z level=info msg="finished complete graph evaluation" controller_path=/ controller_id=pod_logs.feature trace_id=19121e540c1f0a565467f45cd5bef006 duration=2.058606ms ts=2024-12-29T14:20:50.629727204Z level=info msg="finished node evaluation" controller_path=/ controller_id="" trace_id=c64a29d79689717a4f787fda6cccc0e9 node_id=pod_logs.feature duration=2.130446ms ts=2024-12-29T14:20:50.629745604Z level=info msg="finished node evaluation" controller_path=/ controller_id="" trace_id=c64a29d79689717a4f787fda6cccc0e9 node_id=tracing duration=9.8µs ts=2024-12-29T14:20:50.629771964Z level=info msg="finished node evaluation" controller_path=/ controller_id="" trace_id=c64a29d79689717a4f787fda6cccc0e9 node_id=declare.node_logs duration=4.6µs ts=2024-12-29T14:20:50.629799324Z level=info msg="finished node evaluation" controller_path=/ controller_id="" trace_id=c64a29d79689717a4f787fda6cccc0e9 node_id=labelstore duration=16.32µs ts=2024-12-29T14:20:50.629809644Z level=info msg="finished complete graph evaluation" controller_path=/ controller_id="" trace_id=c64a29d79689717a4f787fda6cccc0e9 duration=5.731376ms Error: /etc/alloy/config.alloy:72:3: Failed to build component: building component: creating journal reader: failed to open journal in directory "/var/log/journal": no such file or directory 71 | 72 | loki.source.journal "worker" { | ___^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 73 | | path = "/var/log/journal" 74 | | format_as_json = false 75 | | max_age = "8h" 76 | | relabel_rules = loki.relabel.journal.rules 77 | | labels = { 78 | | job = "integrations/kubernetes/journal", 79 | | instance = sys.env("HOSTNAME"), 80 | | } 81 | | forward_to = [loki.process.journal_logs.receiver] 82 | | } | |_^^^ 83 | interrupt received ts=2024-12-29T14:20:50.630053005Z level=error msg="failed to start reporter" err="context canceled" ts=2024-12-29T14:20:50.630122045Z level=info msg="starting complete graph evaluation" controller_path=/ controller_id=node_logs.feature trace_id=00000000000000000000000000000000 ts=2024-12-29T14:20:50.630178925Z level=info msg="finished node evaluation" controller_path=/ controller_id=node_logs.feature trace_id=00000000000000000000000000000000 node_id=argument.logs_destinations duration=11.76µs ts=2024-12-29T14:20:50.630257285Z level=info msg="starting complete graph evaluation" controller_path=/ controller_id=pod_logs.feature trace_id=00000000000000000000000000000000 ts=2024-12-29T14:20:50.630372045Z level=info msg="finished node evaluation" controller_path=/ controller_id=pod_logs.feature trace_id=00000000000000000000000000000000 node_id=discovery.kubernetes.pods duration=68.08µs ts=2024-12-29T14:20:50.630385485Z level=info msg="finished node evaluation" controller_path=/ controller_id=node_logs.feature trace_id=00000000000000000000000000000000 node_id=loki.process.journal_logs duration=176.04µs ts=2024-12-29T14:20:50.630605846Z level=info msg="finished node evaluation" controller_path=/ controller_id=node_logs.feature trace_id=00000000000000000000000000000000 node_id=loki.relabel.journal duration=149.08µs ts=2024-12-29T14:20:50.630790767Z level=info msg="finished node evaluation" controller_path=/ controller_id=node_logs.feature trace_id=00000000000000000000000000000000 node_id=loki.source.journal.worker duration=128.641µs ts=2024-12-29T14:20:50.630866687Z level=info msg="finished complete graph evaluation" controller_path=/ controller_id=node_logs.feature trace_id=00000000000000000000000000000000 duration=840.363µs ts=2024-12-29T14:20:50.630816007Z level=info msg="finished node evaluation" controller_path=/ controller_id=pod_logs.feature trace_id=00000000000000000000000000000000 node_id=discovery.relabel.filtered_pods duration=422.082µs panic: duplicate metrics collector registration attempted goroutine 142 [running]: github.com/prometheus/client_golang/prometheus.(*wrappingRegisterer).MustRegister(0x40037e0fc0, {0x4003606d40?, 0x0?, 0x0?}) /go/pkg/mod/github.com/prometheus/client_golang@v1.20.5/prometheus/wrap.go:104 +0x14c github.com/grafana/alloy/internal/component/loki/source/journal/internal/target.NewMetrics({0xb4c5728, 0x40037e0fc0}) /src/alloy/internal/component/loki/source/journal/internal/target/metrics.go:33 +0x2a8 github.com/grafana/alloy/internal/component/loki/source/journal.New({{0xffff6571a218, 0x40037e0ea0}, {0x400377ec30, 0x2c}, {0xb439780, 0x40037f00f0}, {0x40037ae2c0, 0x37}, 0x40037d7080, {0xb4c5728, ...}, ...}, ...) /src/alloy/internal/component/loki/source/journal/journal.go:67 +0xfc github.com/grafana/alloy/internal/component/loki/source/journal.init.0.func1({{0xffff6571a218, 0x40037e0ea0}, {0x400377ec30, 0x2c}, {0xb439780, 0x40037f00f0}, {0x40037ae2c0, 0x37}, 0x40037d7080, {0xb4c5728, ...}, ...}, ...) /src/alloy/internal/component/loki/source/journal/journal.go:31 +0x118 github.com/grafana/alloy/internal/runtime/internal/controller.(*BuiltinComponentNode).evaluate(0x40037de488, 0x400307f570) /src/alloy/internal/runtime/internal/controller/node_builtin_component.go:275 +0x3a0 github.com/grafana/alloy/internal/runtime/internal/controller.(*BuiltinComponentNode).Evaluate(0x40037de488, 0x3410980?) /src/alloy/internal/runtime/internal/controller/node_builtin_component.go:248 +0x20 github.com/grafana/alloy/internal/runtime/internal/controller.(*Loader).evaluate(0x40037c9d40, {0xb439780, 0x4003601ae0}, {0xb4f0538, 0x40037de488}) /src/alloy/internal/runtime/internal/controller/loader.go:837 +0x50 github.com/grafana/alloy/internal/runtime/internal/controller.(*Loader).Apply.func2({0xffff6571a238, 0x40037de488}) /src/alloy/internal/runtime/internal/controller/loader.go:207 +0xd34 github.com/grafana/alloy/internal/runtime/internal/dag.WalkTopological(0x4003606680, {0x4003606880, 0x2, 0x10a9ff00?}, 0x40036777e0) /src/alloy/internal/runtime/internal/dag/walk.go:83 +0x194 github.com/grafana/alloy/internal/runtime/internal/controller.(*Loader).Apply(0x40037c9d40, {0x4003614cc0, {0x4003606660, 0x3, 0x4}, {0x4003046280, 0x1, 0x1}, {0x0, 0x0, ...}, ...}) /src/alloy/internal/runtime/internal/controller/loader.go:190 +0x894 github.com/grafana/alloy/internal/runtime.(*Runtime).applyLoaderConfig(0x4003775900, {0x4003614cc0, {0x4003606660, 0x3, 0x4}, {0x4003046280, 0x1, 0x1}, {0x0, 0x0, ...}, ...}) /src/alloy/internal/runtime/alloy.go:334 +0x108 github.com/grafana/alloy/internal/runtime.(*Runtime).loadSource(0x4003775900, 0x400307d8f0, 0x4003614cc0, 0x4003795640) /src/alloy/internal/runtime/alloy.go:320 +0xd8 github.com/grafana/alloy/internal/runtime.(*module).LoadBody(0x40037d6fa0, {0x4003794880?, 0x4003739f45?, 0x4003046278?}, 0x4003614cc0, 0x4003795640) /src/alloy/internal/runtime/module.go:172 +0x6c github.com/grafana/alloy/internal/runtime/internal/controller.(*CustomComponentNode).evaluate(0x400358fba0, 0x400307edd0) /src/alloy/internal/runtime/internal/controller/node_custom_component.go:205 +0x2d0 github.com/grafana/alloy/internal/runtime/internal/controller.(*CustomComponentNode).Evaluate(0x400358fba0, 0x9936d00?) /src/alloy/internal/runtime/internal/controller/node_custom_component.go:167 +0x20 github.com/grafana/alloy/internal/runtime/internal/controller.(*Loader).concurrentEvalFn(0x400358fa00, {0xb43c020, 0x400358fba0}, {0xb4ef448, 0x40036146c0}, {0xb464cc8, 0x4003614570}, 0x40037cf290) /src/alloy/internal/runtime/internal/controller/loader.go:801 +0x520 github.com/grafana/alloy/internal/runtime/internal/controller.(*Loader).EvaluateDependants.func2() /src/alloy/internal/runtime/internal/controller/loader.go:738 +0x3c github.com/grafana/alloy/internal/runtime/internal/worker.(*workQueue).emitNextTask.func1() /src/alloy/internal/runtime/internal/worker/worker_pool.go:181 +0x6c github.com/grafana/alloy/internal/runtime/internal/worker.(*fixedWorkerPool).start.func1() /src/alloy/internal/runtime/internal/worker/worker_pool.go:87 +0x68 created by github.com/grafana/alloy/internal/runtime/internal/worker.(*fixedWorkerPool).start in goroutine 1 /src/alloy/internal/runtime/internal/worker/worker_pool.go:80 +0x2c
Config file:
cluster: name: ${var.cluster_name} destinations: - name: prometheus type: prometheus url: http://prometheus-server.${kubernetes_namespace.cluster_utilities_namespace.id}.svc.cluster.local/api/v1/write tls: insecure_skip_verify: true - name: loki type: loki url: http://loki-gateway.${kubernetes_namespace.cluster_utilities_namespace.id}.svc.cluster.local/loki/api/v1/push tls: insecure_skip_verify: true - name: tempo type: otlp url: http://tempo.${kubernetes_namespace.cluster_utilities_namespace.id}.svc.cluster.local:4317 metrics: enabled: true logs: enabled: true traces: enabled: true tls: insecure_skip_verify: true # Features autoInstrumentation: enabled: true clusterEvents: enabled: true clusterMetrics: enabled: true controlPlane: enabled: true kubeDNS: enabled: true kubeScheduler: enabled: true windows-exporter: deploy: false podLogs: enabled: true nodeLogs: enabled: true integrations: loki: instances: - name: loki labelSelectors: app.kubernetes.io/name: loki logs: enabled: true etcd: instances: - name: k8s-controlplane-etcd labelSelectors: app.kubernetes.io/component: etcd # Collectors applicationObservability: enabled: true receivers: otlp: grpc: enabled: true alloy-metrics: enabled: true alloy: storagePath: /var/lib/alloy mounts: extra: - name: alloy-wal mountPath: /var/lib/alloy controller: enableStatefulSetAutoDeletePVC: true volumeClaimTemplates: - metadata: name: alloy-wal spec: accessModes: ["ReadWriteOnce"] storageClassName: "oci-bv" resources: requests: storage: 5Gi alloy-logs: enabled: true alloy-singleton: enabled: true # An Alloy instance for opening receivers to collect application data. alloy-receiver: # -- Deploy the Alloy instance for opening receivers to collect application data. # @section -- Collectors - Alloy Receiver enabled: true alloy: extraPorts: - name: otlp-grpc port: 4317 targetPort: 4317 protocol: TCP
The text was updated successfully, but these errors were encountered:
No branches or pull requests
Pods are stuck in CrashLoopBackOff:
Logs:
Config file:
The text was updated successfully, but these errors were encountered: