Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix clearing instance metrics when namespace is deleted #1024

Merged
merged 1 commit into from
Aug 29, 2024

Conversation

pavolloffay
Copy link
Collaborator

@pavolloffay pavolloffay commented Aug 29, 2024

Related to #1019

The reconciliation loop was run after the deletion webohok causing the instance metric to be set as failed.

-->Reconciling tempostack chainsaw-loyal-drake/simplest
instance is found, running reconciliation
running build in cert management
running create or update
{"level":"error","ts":"2024-08-29T09:41:44.80454309Z","logger":"tempostack-reconcile","msg":"failed to configure resource","controller":"tempostack","controllerGroup":"tempo.grafana.com","controllerKind":"TempoStack","TempoStack":{"name":"simplest","namespace":"chainsaw-loyal-drake"},"namespace":"chainsaw-loyal-drake","name":"simplest","reconcileID":"546c572e-a89f-4960-9e6f-6adaa4d6bdb2","objectName":"tempo-simplest","objectKind":"/, Kind=","error":"configmaps \"tempo-simplest\" is forbidden: unable to create new content in namespace chainsaw-loyal-drake because it is being terminated","stacktrace":"github.com/grafana/tempo-operator/controllers/tempo.reconcileManagedObjects\n\t/workspace/controllers/tempo/common.go:84\ngithub.com/grafana/tempo-operator/controllers/tempo.(*TempoStackReconciler).createOrUpdate\n\t/workspace/controllers/tempo/tempostack_create_or_update.go:78\ngithub.com/grafana/tempo-operator/controllers/tempo.(*TempoStackReconciler).Reconcile\n\t/workspace/controllers/tempo/tempostack_controller.go:133\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.3/pkg/internal/controller/controller.go:119\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.3/pkg/internal/controller/controller.go:316\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.3/pkg/internal/controller/controller.go:266\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.3/pkg/internal/controller/controller.go:227"}
create or update error failed to create objects for simplest: configmaps "tempo-simplest" is forbidden: unable to create new content in namespace chainsaw-loyal-drake because it is being terminated
{"level":"error","ts":"2024-08-29T09:41:44.884563417Z","msg":"Reconciler error","controller":"tempostack","controllerGroup":"tempo.grafana.com","controllerKind":"TempoStack","TempoStack":{"name":"simplest","namespace":"chainsaw-loyal-drake"},"namespace":"chainsaw-loyal-drake","name":"simplest","reconcileID":"546c572e-a89f-4960-9e6f-6adaa4d6bdb2","error":"failed to create objects for simplest: configmaps \"tempo-simplest\" is forbidden: unable to create new content in namespace chainsaw-loyal-drake because it is being terminated","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.3/pkg/internal/controller/controller.go:329\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.3/pkg/internal/controller/controller.go:266\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.3/pkg/internal/controller/controller.go:227"}

-->Reconciling tempostack chainsaw-loyal-drake/simplest
instance is found, running reconciliation
running build in cert management
{"level":"error","ts":"2024-08-29T09:41:44.886623001Z","logger":"tempostack-reconcile","msg":"failed to configure resource","controller":"tempostack","controllerGroup":"tempo.grafana.com","controllerKind":"TempoStack","TempoStack":{"name":"simplest","namespace":"chainsaw-loyal-drake"},"namespace":"chainsaw-loyal-drake","name":"simplest","reconcileID":"a944d820-ecb1-45c4-83a5-f2cde9dfb99b","tempostacks":"chainsaw-loyal-drake/simplest","event":"createOrRotateCerts","object_name":"tempo-simplest-ca-bundle","object_kind":"&TypeMeta{Kind:,APIVersion:,}","error":"configmaps \"tempo-simplest-ca-bundle\" is forbidden: unable to create new content in namespace chainsaw-loyal-drake because it is being terminated","stacktrace":"github.com/grafana/tempo-operator/internal/certrotation/handlers.CreateOrRotateCertificates\n\t/workspace/internal/certrotation/handlers/rotate_certs.go:78\ngithub.com/grafana/tempo-operator/controllers/tempo.(*TempoStackReconciler).Reconcile\n\t/workspace/controllers/tempo/tempostack_controller.go:125\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.3/pkg/internal/controller/controller.go:119\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.3/pkg/internal/controller/controller.go:316\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.3/pkg/internal/controller/controller.go:266\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.3/pkg/internal/controller/controller.go:227"}
error creating or rotate certificates failed to create or rotate TempoStack certificates
{"level":"error","ts":"2024-08-29T09:41:44.908728513Z","msg":"Reconciler error","controller":"tempostack","controllerGroup":"tempo.grafana.com","controllerKind":"TempoStack","TempoStack":{"name":"simplest","namespace":"chainsaw-loyal-drake"},"namespace":"chainsaw-loyal-drake","name":"simplest","reconcileID":"a944d820-ecb1-45c4-83a5-f2cde9dfb99b","error":"built in cert manager error: failed to create or rotate TempoStack certificates","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.3/pkg/internal/controller/controller.go:329\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.3/pkg/internal/controller/controller.go:266\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.3/pkg/internal/controller/controller.go:227"}

-->Reconciling tempostack chainsaw-loyal-drake/simplest
instance is found, running reconciliation
running build in cert management
{"level":"error","ts":"2024-08-29T09:41:44.911132343Z","logger":"tempostack-reconcile","msg":"failed to configure resource","controller":"tempostack","controllerGroup":"tempo.grafana.com","controllerKind":"TempoStack","TempoStack":{"name":"simplest","namespace":"chainsaw-loyal-drake"},"namespace":"chainsaw-loyal-drake","name":"simplest","reconcileID":"a9ed6fbb-d457-4d13-8288-c55b07089611","tempostacks":"chainsaw-loyal-drake/simplest","event":"createOrRotateCerts","object_name":"tempo-simplest-ca-bundle","object_kind":"&TypeMeta{Kind:,APIVersion:,}","error":"configmaps \"tempo-simplest-ca-bundle\" is forbidden: unable to create new content in namespace chainsaw-loyal-drake because it is being terminated","stacktrace":"github.com/grafana/tempo-operator/internal/certrotation/handlers.CreateOrRotateCertificates\n\t/workspace/internal/certrotation/handlers/rotate_certs.go:78\ngithub.com/grafana/tempo-operator/controllers/tempo.(*TempoStackReconciler).Reconcile\n\t/workspace/controllers/tempo/tempostack_controller.go:125\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.3/pkg/internal/controller/controller.go:119\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.3/pkg/internal/controller/controller.go:316\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.3/pkg/internal/controller/controller.go:266\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.3/pkg/internal/controller/controller.go:227"}
error creating or rotate certificates failed to create or rotate TempoStack certificates
{"level":"error","ts":"2024-08-29T09:41:44.920784312Z","msg":"Reconciler error","controller":"tempostack","controllerGroup":"tempo.grafana.com","controllerKind":"TempoStack","TempoStack":{"name":"simplest","namespace":"chainsaw-loyal-drake"},"namespace":"chainsaw-loyal-drake","name":"simplest","reconcileID":"a9ed6fbb-d457-4d13-8288-c55b07089611","error":"built in cert manager error: failed to create or rotate TempoStack certificates","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.3/pkg/internal/controller/controller.go:329\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.3/pkg/internal/controller/controller.go:266\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.3/pkg/internal/controller/controller.go:227"}

-->Reconciling tempostack chainsaw-loyal-drake/simplest
instance is found, running reconciliation
running build in cert management
{"level":"error","ts":"2024-08-29T09:41:44.923598451Z","logger":"tempostack-reconcile","msg":"failed to configure resource","controller":"tempostack","controllerGroup":"tempo.grafana.com","controllerKind":"TempoStack","TempoStack":{"name":"simplest","namespace":"chainsaw-loyal-drake"},"namespace":"chainsaw-loyal-drake","name":"simplest","reconcileID":"b5dac258-688f-468b-8ac7-8b0971b03996","tempostacks":"chainsaw-loyal-drake/simplest","event":"createOrRotateCerts","object_name":"tempo-simplest-ca-bundle","object_kind":"&TypeMeta{Kind:,APIVersion:,}","error":"configmaps \"tempo-simplest-ca-bundle\" is forbidden: unable to create new content in namespace chainsaw-loyal-drake because it is being terminated","stacktrace":"github.com/grafana/tempo-operator/internal/certrotation/handlers.CreateOrRotateCertificates\n\t/workspace/internal/certrotation/handlers/rotate_certs.go:78\ngithub.com/grafana/tempo-operator/controllers/tempo.(*TempoStackReconciler).Reconcile\n\t/workspace/controllers/tempo/tempostack_controller.go:125\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.3/pkg/internal/controller/controller.go:119\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.3/pkg/internal/controller/controller.go:316\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.3/pkg/internal/controller/controller.go:266\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.3/pkg/internal/controller/controller.go:227"}
error creating or rotate certificates failed to create or rotate TempoStack certificates
{"level":"error","ts":"2024-08-29T09:41:44.9326483Z","msg":"Reconciler error","controller":"tempostack","controllerGroup":"tempo.grafana.com","controllerKind":"TempoStack","TempoStack":{"name":"simplest","namespace":"chainsaw-loyal-drake"},"namespace":"chainsaw-loyal-drake","name":"simplest","reconcileID":"b5dac258-688f-468b-8ac7-8b0971b03996","error":"built in cert manager error: failed to create or rotate TempoStack certificates","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.3/pkg/internal/controller/controller.go:329\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.3/pkg/internal/controller/controller.go:266\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.3/pkg/internal/controller/controller.go:227"}

-->Reconciling tempostack chainsaw-loyal-drake/simplest
instance is found, running reconciliation
running build in cert management
{"level":"error","ts":"2024-08-29T09:41:44.943632907Z","logger":"tempostack-reconcile","msg":"failed to configure resource","controller":"tempostack","controllerGroup":"tempo.grafana.com","controllerKind":"TempoStack","TempoStack":{"name":"simplest","namespace":"chainsaw-loyal-drake"},"namespace":"chainsaw-loyal-drake","name":"simplest","reconcileID":"0e75583c-028c-46b3-9616-3c0235ba2cf2","tempostacks":"chainsaw-loyal-drake/simplest","event":"createOrRotateCerts","object_name":"tempo-simplest-ca-bundle","object_kind":"&TypeMeta{Kind:,APIVersion:,}","error":"configmaps \"tempo-simplest-ca-bundle\" is forbidden: unable to create new content in namespace chainsaw-loyal-drake because it is being terminated","stacktrace":"github.com/grafana/tempo-operator/internal/certrotation/handlers.CreateOrRotateCertificates\n\t/workspace/internal/certrotation/handlers/rotate_certs.go:78\ngithub.com/grafana/tempo-operator/controllers/tempo.(*TempoStackReconciler).Reconcile\n\t/workspace/controllers/tempo/tempostack_controller.go:125\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.3/pkg/internal/controller/controller.go:119\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.3/pkg/internal/controller/controller.go:316\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.3/pkg/internal/controller/controller.go:266\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.3/pkg/internal/controller/controller.go:227"}
error creating or rotate certificates failed to create or rotate TempoStack certificates
{"level":"error","ts":"2024-08-29T09:41:44.956636985Z","msg":"Reconciler error","controller":"tempostack","controllerGroup":"tempo.grafana.com","controllerKind":"TempoStack","TempoStack":{"name":"simplest","namespace":"chainsaw-loyal-drake"},"namespace":"chainsaw-loyal-drake","name":"simplest","reconcileID":"0e75583c-028c-46b3-9616-3c0235ba2cf2","error":"built in cert manager error: failed to create or rotate TempoStack certificates","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.3/pkg/internal/controller/controller.go:329\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.3/pkg/internal/controller/controller.go:266\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.3/pkg/internal/controller/controller.go:227"}

-->Reconciling tempostack chainsaw-loyal-drake/simplest
instance is found, running reconciliation
running build in cert management
{"level":"error","ts":"2024-08-29T09:41:45.04003063Z","logger":"tempostack-reconcile","msg":"failed to configure resource","controller":"tempostack","controllerGroup":"tempo.grafana.com","controllerKind":"TempoStack","TempoStack":{"name":"simplest","namespace":"chainsaw-loyal-drake"},"namespace":"chainsaw-loyal-drake","name":"simplest","reconcileID":"54110e40-726f-4473-a2a7-456335daad9f","tempostacks":"chainsaw-loyal-drake/simplest","event":"createOrRotateCerts","object_name":"tempo-simplest-ca-bundle","object_kind":"&TypeMeta{Kind:,APIVersion:,}","error":"configmaps \"tempo-simplest-ca-bundle\" is forbidden: unable to create new content in namespace chainsaw-loyal-drake because it is being terminated","stacktrace":"github.com/grafana/tempo-operator/internal/certrotation/handlers.CreateOrRotateCertificates\n\t/workspace/internal/certrotation/handlers/rotate_certs.go:78\ngithub.com/grafana/tempo-operator/controllers/tempo.(*TempoStackReconciler).Reconcile\n\t/workspace/controllers/tempo/tempostack_controller.go:125\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.3/pkg/internal/controller/controller.go:119\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.3/pkg/internal/controller/controller.go:316\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.3/pkg/internal/controller/controller.go:266\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.3/pkg/internal/controller/controller.go:227"}
error creating or rotate certificates failed to create or rotate TempoStack certificates
{"level":"error","ts":"2024-08-29T09:41:45.054945442Z","msg":"Reconciler error","controller":"tempostack","controllerGroup":"tempo.grafana.com","controllerKind":"TempoStack","TempoStack":{"name":"simplest","namespace":"chainsaw-loyal-drake"},"namespace":"chainsaw-loyal-drake","name":"simplest","reconcileID":"54110e40-726f-4473-a2a7-456335daad9f","error":"built in cert manager error: failed to create or rotate TempoStack certificates","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.3/pkg/internal/controller/controller.go:329\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.3/pkg/internal/controller/controller.go:266\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.3/pkg/internal/controller/controller.go:227"}

-->Reconciling tempostack chainsaw-loyal-drake/simplest
instance is found, running reconciliation
running build in cert management
{"level":"error","ts":"2024-08-29T09:41:45.217640355Z","logger":"tempostack-reconcile","msg":"failed to configure resource","controller":"tempostack","controllerGroup":"tempo.grafana.com","controllerKind":"TempoStack","TempoStack":{"name":"simplest","namespace":"chainsaw-loyal-drake"},"namespace":"chainsaw-loyal-drake","name":"simplest","reconcileID":"818703ff-42da-4c4a-90a2-dcfd1713fd7d","tempostacks":"chainsaw-loyal-drake/simplest","event":"createOrRotateCerts","object_name":"tempo-simplest-ca-bundle","object_kind":"&TypeMeta{Kind:,APIVersion:,}","error":"configmaps \"tempo-simplest-ca-bundle\" is forbidden: unable to create new content in namespace chainsaw-loyal-drake because it is being terminated","stacktrace":"github.com/grafana/tempo-operator/internal/certrotation/handlers.CreateOrRotateCertificates\n\t/workspace/internal/certrotation/handlers/rotate_certs.go:78\ngithub.com/grafana/tempo-operator/controllers/tempo.(*TempoStackReconciler).Reconcile\n\t/workspace/controllers/tempo/tempostack_controller.go:125\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.3/pkg/internal/controller/controller.go:119\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.3/pkg/internal/controller/controller.go:316\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.3/pkg/internal/controller/controller.go:266\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.3/pkg/internal/controller/controller.go:227"}
error creating or rotate certificates failed to create or rotate TempoStack certificates
{"level":"error","ts":"2024-08-29T09:41:45.244609271Z","msg":"Reconciler error","controller":"tempostack","controllerGroup":"tempo.grafana.com","controllerKind":"TempoStack","TempoStack":{"name":"simplest","namespace":"chainsaw-loyal-drake"},"namespace":"chainsaw-loyal-drake","name":"simplest","reconcileID":"818703ff-42da-4c4a-90a2-dcfd1713fd7d","error":"built in cert manager error: failed to create or rotate TempoStack certificates","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.3/pkg/internal/controller/controller.go:329\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.3/pkg/internal/controller/controller.go:266\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.3/pkg/internal/controller/controller.go:227"}

-->Reconciling tempostack chainsaw-loyal-drake/simplest
instance is found, running reconciliation
running build in cert management
{"level":"error","ts":"2024-08-29T09:41:45.568455632Z","logger":"tempostack-reconcile","msg":"failed to configure resource","controller":"tempostack","controllerGroup":"tempo.grafana.com","controllerKind":"TempoStack","TempoStack":{"name":"simplest","namespace":"chainsaw-loyal-drake"},"namespace":"chainsaw-loyal-drake","name":"simplest","reconcileID":"ce344116-39bb-4ecd-95c8-958107bbc14d","tempostacks":"chainsaw-loyal-drake/simplest","event":"createOrRotateCerts","object_name":"tempo-simplest-ca-bundle","object_kind":"&TypeMeta{Kind:,APIVersion:,}","error":"configmaps \"tempo-simplest-ca-bundle\" is forbidden: unable to create new content in namespace chainsaw-loyal-drake because it is being terminated","stacktrace":"github.com/grafana/tempo-operator/internal/certrotation/handlers.CreateOrRotateCertificates\n\t/workspace/internal/certrotation/handlers/rotate_certs.go:78\ngithub.com/grafana/tempo-operator/controllers/tempo.(*TempoStackReconciler).Reconcile\n\t/workspace/controllers/tempo/tempostack_controller.go:125\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.3/pkg/internal/controller/controller.go:119\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.3/pkg/internal/controller/controller.go:316\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.3/pkg/internal/controller/controller.go:266\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.3/pkg/internal/controller/controller.go:227"}
error creating or rotate certificates failed to create or rotate TempoStack certificates
{"level":"error","ts":"2024-08-29T09:41:45.591105486Z","msg":"Reconciler error","controller":"tempostack","controllerGroup":"tempo.grafana.com","controllerKind":"TempoStack","TempoStack":{"name":"simplest","namespace":"chainsaw-loyal-drake"},"namespace":"chainsaw-loyal-drake","name":"simplest","reconcileID":"ce344116-39bb-4ecd-95c8-958107bbc14d","error":"built in cert manager error: failed to create or rotate TempoStack certificates","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.3/pkg/internal/controller/controller.go:329\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.3/pkg/internal/controller/controller.go:266\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.3/pkg/internal/controller/controller.go:227"}

-->Reconciling tempostack chainsaw-loyal-drake/simplest
instance is found, running reconciliation
running build in cert management
{"level":"error","ts":"2024-08-29T09:41:45.759866099Z","logger":"tempostack-reconcile","msg":"failed to configure resource","controller":"tempostack","controllerGroup":"tempo.grafana.com","controllerKind":"TempoStack","TempoStack":{"name":"simplest","namespace":"chainsaw-loyal-drake"},"namespace":"chainsaw-loyal-drake","name":"simplest","reconcileID":"f659dad9-5b7d-48e9-a494-77d4f9d9e832","tempostacks":"chainsaw-loyal-drake/simplest","event":"createOrRotateCerts","object_name":"tempo-simplest-ca-bundle","object_kind":"&TypeMeta{Kind:,APIVersion:,}","error":"configmaps \"tempo-simplest-ca-bundle\" is forbidden: unable to create new content in namespace chainsaw-loyal-drake because it is being terminated","stacktrace":"github.com/grafana/tempo-operator/internal/certrotation/handlers.CreateOrRotateCertificates\n\t/workspace/internal/certrotation/handlers/rotate_certs.go:78\ngithub.com/grafana/tempo-operator/controllers/tempo.(*TempoStackReconciler).Reconcile\n\t/workspace/controllers/tempo/tempostack_controller.go:125\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.3/pkg/internal/controller/controller.go:119\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.3/pkg/internal/controller/controller.go:316\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.3/pkg/internal/controller/controller.go:266\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.3/pkg/internal/controller/controller.go:227"}
error creating or rotate certificates failed to create or rotate TempoStack certificates
{"level":"error","ts":"2024-08-29T09:41:45.772825122Z","msg":"Reconciler error","controller":"tempostack","controllerGroup":"tempo.grafana.com","controllerKind":"TempoStack","TempoStack":{"name":"simplest","namespace":"chainsaw-loyal-drake"},"namespace":"chainsaw-loyal-drake","name":"simplest","reconcileID":"f659dad9-5b7d-48e9-a494-77d4f9d9e832","error":"built in cert manager error: failed to create or rotate TempoStack certificates","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.3/pkg/internal/controller/controller.go:329\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.3/pkg/internal/controller/controller.go:266\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.3/pkg/internal/controller/controller.go:227"}

-->Reconciling tempostack chainsaw-loyal-drake/simplest
instance is found, running reconciliation
running build in cert management
{"level":"error","ts":"2024-08-29T09:41:45.990902279Z","logger":"tempostack-reconcile","msg":"failed to configure resource","controller":"tempostack","controllerGroup":"tempo.grafana.com","controllerKind":"TempoStack","TempoStack":{"name":"simplest","namespace":"chainsaw-loyal-drake"},"namespace":"chainsaw-loyal-drake","name":"simplest","reconcileID":"7445b22d-1f8c-4bbe-90ee-06de160559d7","tempostacks":"chainsaw-loyal-drake/simplest","event":"createOrRotateCerts","object_name":"tempo-simplest-ca-bundle","object_kind":"&TypeMeta{Kind:,APIVersion:,}","error":"configmaps \"tempo-simplest-ca-bundle\" is forbidden: unable to create new content in namespace chainsaw-loyal-drake because it is being terminated","stacktrace":"github.com/grafana/tempo-operator/internal/certrotation/handlers.CreateOrRotateCertificates\n\t/workspace/internal/certrotation/handlers/rotate_certs.go:78\ngithub.com/grafana/tempo-operator/controllers/tempo.(*TempoStackReconciler).Reconcile\n\t/workspace/controllers/tempo/tempostack_controller.go:125\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.3/pkg/internal/controller/controller.go:119\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.3/pkg/internal/controller/controller.go:316\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.3/pkg/internal/controller/controller.go:266\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.3/pkg/internal/controller/controller.go:227"}
error creating or rotate certificates failed to create or rotate TempoStack certificates
{"level":"error","ts":"2024-08-29T09:41:46.013553615Z","msg":"Reconciler error","controller":"tempostack","controllerGroup":"tempo.grafana.com","controllerKind":"TempoStack","TempoStack":{"name":"simplest","namespace":"chainsaw-loyal-drake"},"namespace":"chainsaw-loyal-drake","name":"simplest","reconcileID":"7445b22d-1f8c-4bbe-90ee-06de160559d7","error":"built in cert manager error: failed to create or rotate TempoStack certificates","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.3/pkg/internal/controller/controller.go:329\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.3/pkg/internal/controller/controller.go:266\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.3/pkg/internal/controller/controller.go:227"}

-->Reconciling tempostack chainsaw-loyal-drake/simplest
instance is found, running reconciliation
running build in cert management
{"level":"error","ts":"2024-08-29T09:41:46.204762724Z","logger":"tempostack-reconcile","msg":"failed to configure resource","controller":"tempostack","controllerGroup":"tempo.grafana.com","controllerKind":"TempoStack","TempoStack":{"name":"simplest","namespace":"chainsaw-loyal-drake"},"namespace":"chainsaw-loyal-drake","name":"simplest","reconcileID":"6676fb07-9628-453d-afb5-8749b4c4f1c0","tempostacks":"chainsaw-loyal-drake/simplest","event":"createOrRotateCerts","object_name":"tempo-simplest-ca-bundle","object_kind":"&TypeMeta{Kind:,APIVersion:,}","error":"configmaps \"tempo-simplest-ca-bundle\" is forbidden: unable to create new content in namespace chainsaw-loyal-drake because it is being terminated","stacktrace":"github.com/grafana/tempo-operator/internal/certrotation/handlers.CreateOrRotateCertificates\n\t/workspace/internal/certrotation/handlers/rotate_certs.go:78\ngithub.com/grafana/tempo-operator/controllers/tempo.(*TempoStackReconciler).Reconcile\n\t/workspace/controllers/tempo/tempostack_controller.go:125\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.3/pkg/internal/controller/controller.go:119\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.3/pkg/internal/controller/controller.go:316\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.3/pkg/internal/controller/controller.go:266\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.3/pkg/internal/controller/controller.go:227"}
error creating or rotate certificates failed to create or rotate TempoStack certificates
{"level":"error","ts":"2024-08-29T09:41:46.221985526Z","msg":"Reconciler error","controller":"tempostack","controllerGroup":"tempo.grafana.com","controllerKind":"TempoStack","TempoStack":{"name":"simplest","namespace":"chainsaw-loyal-drake"},"namespace":"chainsaw-loyal-drake","name":"simplest","reconcileID":"6676fb07-9628-453d-afb5-8749b4c4f1c0","error":"built in cert manager error: failed to create or rotate TempoStack certificates","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.3/pkg/internal/controller/controller.go:329\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.3/pkg/internal/controller/controller.go:266\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.3/pkg/internal/controller/controller.go:227"}

-->Reconciling tempostack chainsaw-loyal-drake/simplest
instance is found, running reconciliation
running build in cert management
{"level":"error","ts":"2024-08-29T09:41:46.325294554Z","logger":"tempostack-reconcile","msg":"failed to configure resource","controller":"tempostack","controllerGroup":"tempo.grafana.com","controllerKind":"TempoStack","TempoStack":{"name":"simplest","namespace":"chainsaw-loyal-drake"},"namespace":"chainsaw-loyal-drake","name":"simplest","reconcileID":"39a865e7-e584-4283-81d9-cbe9a5539c8b","tempostacks":"chainsaw-loyal-drake/simplest","event":"createOrRotateCerts","object_name":"tempo-simplest-signing-ca","object_kind":"&TypeMeta{Kind:,APIVersion:,}","error":"secrets \"tempo-simplest-signing-ca\" is forbidden: unable to create new content in namespace chainsaw-loyal-drake because it is being terminated","stacktrace":"github.com/grafana/tempo-operator/internal/certrotation/handlers.CreateOrRotateCertificates\n\t/workspace/internal/certrotation/handlers/rotate_certs.go:78\ngithub.com/grafana/tempo-operator/controllers/tempo.(*TempoStackReconciler).Reconcile\n\t/workspace/controllers/tempo/tempostack_controller.go:125\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.3/pkg/internal/controller/controller.go:119\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.3/pkg/internal/controller/controller.go:316\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.3/pkg/internal/controller/controller.go:266\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.3/pkg/internal/controller/controller.go:227"}
{"level":"error","ts":"2024-08-29T09:41:46.326617046Z","logger":"tempostack-reconcile","msg":"failed to configure resource","controller":"tempostack","controllerGroup":"tempo.grafana.com","controllerKind":"TempoStack","TempoStack":{"name":"simplest","namespace":"chainsaw-loyal-drake"},"namespace":"chainsaw-loyal-drake","name":"simplest","reconcileID":"39a865e7-e584-4283-81d9-cbe9a5539c8b","tempostacks":"chainsaw-loyal-drake/simplest","event":"createOrRotateCerts","object_name":"tempo-simplest-ca-bundle","object_kind":"&TypeMeta{Kind:,APIVersion:,}","error":"configmaps \"tempo-simplest-ca-bundle\" is forbidden: unable to create new content in namespace chainsaw-loyal-drake because it is being terminated","stacktrace":"github.com/grafana/tempo-operator/internal/certrotation/handlers.CreateOrRotateCertificates\n\t/workspace/internal/certrotation/handlers/rotate_certs.go:78\ngithub.com/grafana/tempo-operator/controllers/tempo.(*TempoStackReconciler).Reconcile\n\t/workspace/controllers/tempo/tempostack_controller.go:125\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.3/pkg/internal/controller/controller.go:119\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.3/pkg/internal/controller/controller.go:316\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.3/pkg/internal/controller/controller.go:266\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.3/pkg/internal/controller/controller.go:227"}
{"level":"error","ts":"2024-08-29T09:41:46.327672887Z","logger":"tempostack-reconcile","msg":"failed to configure resource","controller":"tempostack","controllerGroup":"tempo.grafana.com","controllerKind":"TempoStack","TempoStack":{"name":"simplest","namespace":"chainsaw-loyal-drake"},"namespace":"chainsaw-loyal-drake","name":"simplest","reconcileID":"39a865e7-e584-4283-81d9-cbe9a5539c8b","tempostacks":"chainsaw-loyal-drake/simplest","event":"createOrRotateCerts","object_name":"tempo-simplest-ingester-mtls","object_kind":"&TypeMeta{Kind:,APIVersion:,}","error":"secrets \"tempo-simplest-ingester-mtls\" is forbidden: unable to create new content in namespace chainsaw-loyal-drake because it is being terminated","stacktrace":"github.com/grafana/tempo-operator/internal/certrotation/handlers.CreateOrRotateCertificates\n\t/workspace/internal/certrotation/handlers/rotate_certs.go:78\ngithub.com/grafana/tempo-operator/controllers/tempo.(*TempoStackReconciler).Reconcile\n\t/workspace/controllers/tempo/tempostack_controller.go:125\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.3/pkg/internal/controller/controller.go:119\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.3/pkg/internal/controller/controller.go:316\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.3/pkg/internal/controller/controller.go:266\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.3/pkg/internal/controller/controller.go:227"}
{"level":"error","ts":"2024-08-29T09:41:46.330892036Z","logger":"tempostack-reconcile","msg":"failed to configure resource","controller":"tempostack","controllerGroup":"tempo.grafana.com","controllerKind":"TempoStack","TempoStack":{"name":"simplest","namespace":"chainsaw-loyal-drake"},"namespace":"chainsaw-loyal-drake","name":"simplest","reconcileID":"39a865e7-e584-4283-81d9-cbe9a5539c8b","tempostacks":"chainsaw-loyal-drake/simplest","event":"createOrRotateCerts","object_name":"tempo-simplest-querier-mtls","object_kind":"&TypeMeta{Kind:,APIVersion:,}","error":"secrets \"tempo-simplest-querier-mtls\" is forbidden: unable to create new content in namespace chainsaw-loyal-drake because it is being terminated","stacktrace":"github.com/grafana/tempo-operator/internal/certrotation/handlers.CreateOrRotateCertificates\n\t/workspace/internal/certrotation/handlers/rotate_certs.go:78\ngithub.com/grafana/tempo-operator/controllers/tempo.(*TempoStackReconciler).Reconcile\n\t/workspace/controllers/tempo/tempostack_controller.go:125\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.3/pkg/internal/controller/controller.go:119\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.3/pkg/internal/controller/controller.go:316\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.3/pkg/internal/controller/controller.go:266\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.3/pkg/internal/controller/controller.go:227"}
{"level":"error","ts":"2024-08-29T09:41:46.33226402Z","logger":"tempostack-reconcile","msg":"failed to configure resource","controller":"tempostack","controllerGroup":"tempo.grafana.com","controllerKind":"TempoStack","TempoStack":{"name":"simplest","namespace":"chainsaw-loyal-drake"},"namespace":"chainsaw-loyal-drake","name":"simplest","reconcileID":"39a865e7-e584-4283-81d9-cbe9a5539c8b","tempostacks":"chainsaw-loyal-drake/simplest","event":"createOrRotateCerts","object_name":"tempo-simplest-query-frontend-mtls","object_kind":"&TypeMeta{Kind:,APIVersion:,}","error":"secrets \"tempo-simplest-query-frontend-mtls\" is forbidden: unable to create new content in namespace chainsaw-loyal-drake because it is being terminated","stacktrace":"github.com/grafana/tempo-operator/internal/certrotation/handlers.CreateOrRotateCertificates\n\t/workspace/internal/certrotation/handlers/rotate_certs.go:78\ngithub.com/grafana/tempo-operator/controllers/tempo.(*TempoStackReconciler).Reconcile\n\t/workspace/controllers/tempo/tempostack_controller.go:125\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.3/pkg/internal/controller/controller.go:119\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.3/pkg/internal/controller/controller.go:316\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.3/pkg/internal/controller/controller.go:266\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.3/pkg/internal/controller/controller.go:227"}
{"level":"error","ts":"2024-08-29T09:41:46.333397086Z","logger":"tempostack-reconcile","msg":"failed to configure resource","controller":"tempostack","controllerGroup":"tempo.grafana.com","controllerKind":"TempoStack","TempoStack":{"name":"simplest","namespace":"chainsaw-loyal-drake"},"namespace":"chainsaw-loyal-drake","name":"simplest","reconcileID":"39a865e7-e584-4283-81d9-cbe9a5539c8b","tempostacks":"chainsaw-loyal-drake/simplest","event":"createOrRotateCerts","object_name":"tempo-simplest-compactor-mtls","object_kind":"&TypeMeta{Kind:,APIVersion:,}","error":"secrets \"tempo-simplest-compactor-mtls\" is forbidden: unable to create new content in namespace chainsaw-loyal-drake because it is being terminated","stacktrace":"github.com/grafana/tempo-operator/internal/certrotation/handlers.CreateOrRotateCertificates\n\t/workspace/internal/certrotation/handlers/rotate_certs.go:78\ngithub.com/grafana/tempo-operator/controllers/tempo.(*TempoStackReconciler).Reconcile\n\t/workspace/controllers/tempo/tempostack_controller.go:125\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.3/pkg/internal/controller/controller.go:119\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.3/pkg/internal/controller/controller.go:316\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.3/pkg/internal/controller/controller.go:266\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.3/pkg/internal/controller/controller.go:227"}
{"level":"error","ts":"2024-08-29T09:41:46.335907255Z","logger":"tempostack-reconcile","msg":"failed to configure resource","controller":"tempostack","controllerGroup":"tempo.grafana.com","controllerKind":"TempoStack","TempoStack":{"name":"simplest","namespace":"chainsaw-loyal-drake"},"namespace":"chainsaw-loyal-drake","name":"simplest","reconcileID":"39a865e7-e584-4283-81d9-cbe9a5539c8b","tempostacks":"chainsaw-loyal-drake/simplest","event":"createOrRotateCerts","object_name":"tempo-simplest-gateway-mtls","object_kind":"&TypeMeta{Kind:,APIVersion:,}","error":"secrets \"tempo-simplest-gateway-mtls\" is forbidden: unable to create new content in namespace chainsaw-loyal-drake because it is being terminated","stacktrace":"github.com/grafana/tempo-operator/internal/certrotation/handlers.CreateOrRotateCertificates\n\t/workspace/internal/certrotation/handlers/rotate_certs.go:78\ngithub.com/grafana/tempo-operator/controllers/tempo.(*TempoStackReconciler).Reconcile\n\t/workspace/controllers/tempo/tempostack_controller.go:125\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.3/pkg/internal/controller/controller.go:119\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.3/pkg/internal/controller/controller.go:316\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.3/pkg/internal/controller/controller.go:266\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.3/pkg/internal/controller/controller.go:227"}
{"level":"error","ts":"2024-08-29T09:41:46.336978796Z","logger":"tempostack-reconcile","msg":"failed to configure resource","controller":"tempostack","controllerGroup":"tempo.grafana.com","controllerKind":"TempoStack","TempoStack":{"name":"simplest","namespace":"chainsaw-loyal-drake"},"namespace":"chainsaw-loyal-drake","name":"simplest","reconcileID":"39a865e7-e584-4283-81d9-cbe9a5539c8b","tempostacks":"chainsaw-loyal-drake/simplest","event":"createOrRotateCerts","object_name":"tempo-simplest-distributor-mtls","object_kind":"&TypeMeta{Kind:,APIVersion:,}","error":"secrets \"tempo-simplest-distributor-mtls\" is forbidden: unable to create new content in namespace chainsaw-loyal-drake because it is being terminated","stacktrace":"github.com/grafana/tempo-operator/internal/certrotation/handlers.CreateOrRotateCertificates\n\t/workspace/internal/certrotation/handlers/rotate_certs.go:78\ngithub.com/grafana/tempo-operator/controllers/tempo.(*TempoStackReconciler).Reconcile\n\t/workspace/controllers/tempo/tempostack_controller.go:125\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.3/pkg/internal/controller/controller.go:119\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.3/pkg/internal/controller/controller.go:316\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.3/pkg/internal/controller/controller.go:266\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.3/pkg/internal/controller/controller.go:227"}
error creating or rotate certificates failed to create or rotate TempoStack certificates
{"level":"error","ts":"2024-08-29T09:41:46.343896764Z","msg":"Reconciler error","controller":"tempostack","controllerGroup":"tempo.grafana.com","controllerKind":"TempoStack","TempoStack":{"name":"simplest","namespace":"chainsaw-loyal-drake"},"namespace":"chainsaw-loyal-drake","name":"simplest","reconcileID":"39a865e7-e584-4283-81d9-cbe9a5539c8b","error":"built in cert manager error: failed to create or rotate TempoStack certificates","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.3/pkg/internal/controller/controller.go:329\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.3/pkg/internal/controller/controller.go:266\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.3/pkg/internal/controller/controller.go:227"}

-->Reconciling tempostack chainsaw-loyal-drake/simplest
instance is found, running reconciliation
running build in cert management
<--- delete webhook
{"level":"info","ts":"2024-08-29T09:41:46.945062798Z","logger":"certrotation-reconcile","msg":"Skipping reconciliation for unmanaged TempoStack resource","controller":"tempostack","controllerGroup":"tempo.grafana.com","controllerKind":"TempoStack","TempoStack":{"name":"simplest","namespace":"chainsaw-loyal-drake"},"namespace":"chainsaw-loyal-drake","name":"simplest","reconcileID":"7de9981d-088e-4c45-97ec-d43b06778ddc","tempo":{"name":"simplest","namespace":"chainsaw-loyal-drake"},"name":"chainsaw-loyal-drake/simplest"}
{"level":"error","ts":"2024-08-29T09:41:47.119842764Z","logger":"tempostack-reconcile","msg":"failed to configure resource","controller":"tempostack","controllerGroup":"tempo.grafana.com","controllerKind":"TempoStack","TempoStack":{"name":"simplest","namespace":"chainsaw-loyal-drake"},"namespace":"chainsaw-loyal-drake","name":"simplest","reconcileID":"cd3b8daf-83ff-49c4-8acd-4ef8359e7803","tempostacks":"chainsaw-loyal-drake/simplest","event":"createOrRotateCerts","object_name":"tempo-simplest-signing-ca","object_kind":"&TypeMeta{Kind:,APIVersion:,}","error":"secrets \"tempo-simplest-signing-ca\" is forbidden: unable to create new content in namespace chainsaw-loyal-drake because it is being terminated","stacktrace":"github.com/grafana/tempo-operator/internal/certrotation/handlers.CreateOrRotateCertificates\n\t/workspace/internal/certrotation/handlers/rotate_certs.go:78\ngithub.com/grafana/tempo-operator/controllers/tempo.(*TempoStackReconciler).Reconcile\n\t/workspace/controllers/tempo/tempostack_controller.go:125\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.3/pkg/internal/controller/controller.go:119\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.3/pkg/internal/controller/controller.go:316\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.3/pkg/internal/controller/controller.go:266\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.3/pkg/internal/controller/controller.go:227"}
{"level":"error","ts":"2024-08-29T09:41:47.12123206Z","logger":"tempostack-reconcile","msg":"failed to configure resource","controller":"tempostack","controllerGroup":"tempo.grafana.com","controllerKind":"TempoStack","TempoStack":{"name":"simplest","namespace":"chainsaw-loyal-drake"},"namespace":"chainsaw-loyal-drake","name":"simplest","reconcileID":"cd3b8daf-83ff-49c4-8acd-4ef8359e7803","tempostacks":"chainsaw-loyal-drake/simplest","event":"createOrRotateCerts","object_name":"tempo-simplest-ca-bundle","object_kind":"&TypeMeta{Kind:,APIVersion:,}","error":"configmaps \"tempo-simplest-ca-bundle\" is forbidden: unable to create new content in namespace chainsaw-loyal-drake because it is being terminated","stacktrace":"github.com/grafana/tempo-operator/internal/certrotation/handlers.CreateOrRotateCertificates\n\t/workspace/internal/certrotation/handlers/rotate_certs.go:78\ngithub.com/grafana/tempo-operator/controllers/tempo.(*TempoStackReconciler).Reconcile\n\t/workspace/controllers/tempo/tempostack_controller.go:125\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.3/pkg/internal/controller/controller.go:119\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.3/pkg/internal/controller/controller.go:316\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.3/pkg/internal/controller/controller.go:266\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.3/pkg/internal/controller/controller.go:227"}
{"level":"error","ts":"2024-08-29T09:41:47.122862429Z","logger":"tempostack-reconcile","msg":"failed to configure resource","controller":"tempostack","controllerGroup":"tempo.grafana.com","controllerKind":"TempoStack","TempoStack":{"name":"simplest","namespace":"chainsaw-loyal-drake"},"namespace":"chainsaw-loyal-drake","name":"simplest","reconcileID":"cd3b8daf-83ff-49c4-8acd-4ef8359e7803","tempostacks":"chainsaw-loyal-drake/simplest","event":"createOrRotateCerts","object_name":"tempo-simplest-ingester-mtls","object_kind":"&TypeMeta{Kind:,APIVersion:,}","error":"secrets \"tempo-simplest-ingester-mtls\" is forbidden: unable to create new content in namespace chainsaw-loyal-drake because it is being terminated","stacktrace":"github.com/grafana/tempo-operator/internal/certrotation/handlers.CreateOrRotateCertificates\n\t/workspace/internal/certrotation/handlers/rotate_certs.go:78\ngithub.com/grafana/tempo-operator/controllers/tempo.(*TempoStackReconciler).Reconcile\n\t/workspace/controllers/tempo/tempostack_controller.go:125\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.3/pkg/internal/controller/controller.go:119\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.3/pkg/internal/controller/controller.go:316\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.3/pkg/internal/controller/controller.go:266\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.3/pkg/internal/controller/controller.go:227"}
{"level":"error","ts":"2024-08-29T09:41:47.125637264Z","logger":"tempostack-reconcile","msg":"failed to configure resource","controller":"tempostack","controllerGroup":"tempo.grafana.com","controllerKind":"TempoStack","TempoStack":{"name":"simplest","namespace":"chainsaw-loyal-drake"},"namespace":"chainsaw-loyal-drake","name":"simplest","reconcileID":"cd3b8daf-83ff-49c4-8acd-4ef8359e7803","tempostacks":"chainsaw-loyal-drake/simplest","event":"createOrRotateCerts","object_name":"tempo-simplest-querier-mtls","object_kind":"&TypeMeta{Kind:,APIVersion:,}","error":"secrets \"tempo-simplest-querier-mtls\" is forbidden: unable to create new content in namespace chainsaw-loyal-drake because it is being terminated","stacktrace":"github.com/grafana/tempo-operator/internal/certrotation/handlers.CreateOrRotateCertificates\n\t/workspace/internal/certrotation/handlers/rotate_certs.go:78\ngithub.com/grafana/tempo-operator/controllers/tempo.(*TempoStackReconciler).Reconcile\n\t/workspace/controllers/tempo/tempostack_controller.go:125\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.3/pkg/internal/controller/controller.go:119\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.3/pkg/internal/controller/controller.go:316\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.3/pkg/internal/controller/controller.go:266\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.3/pkg/internal/controller/controller.go:227"}
{"level":"error","ts":"2024-08-29T09:41:47.128165588Z","logger":"tempostack-reconcile","msg":"failed to configure resource","controller":"tempostack","controllerGroup":"tempo.grafana.com","controllerKind":"TempoStack","TempoStack":{"name":"simplest","namespace":"chainsaw-loyal-drake"},"namespace":"chainsaw-loyal-drake","name":"simplest","reconcileID":"cd3b8daf-83ff-49c4-8acd-4ef8359e7803","tempostacks":"chainsaw-loyal-drake/simplest","event":"createOrRotateCerts","object_name":"tempo-simplest-query-frontend-mtls","object_kind":"&TypeMeta{Kind:,APIVersion:,}","error":"secrets \"tempo-simplest-query-frontend-mtls\" is forbidden: unable to create new content in namespace chainsaw-loyal-drake because it is being terminated","stacktrace":"github.com/grafana/tempo-operator/internal/certrotation/handlers.CreateOrRotateCertificates\n\t/workspace/internal/certrotation/handlers/rotate_certs.go:78\ngithub.com/grafana/tempo-operator/controllers/tempo.(*TempoStackReconciler).Reconcile\n\t/workspace/controllers/tempo/tempostack_controller.go:125\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.3/pkg/internal/controller/controller.go:119\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.3/pkg/internal/controller/controller.go:316\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.3/pkg/internal/controller/controller.go:266\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.3/pkg/internal/controller/controller.go:227"}
{"level":"error","ts":"2024-08-29T09:41:47.13160468Z","logger":"tempostack-reconcile","msg":"failed to configure resource","controller":"tempostack","controllerGroup":"tempo.grafana.com","controllerKind":"TempoStack","TempoStack":{"name":"simplest","namespace":"chainsaw-loyal-drake"},"namespace":"chainsaw-loyal-drake","name":"simplest","reconcileID":"cd3b8daf-83ff-49c4-8acd-4ef8359e7803","tempostacks":"chainsaw-loyal-drake/simplest","event":"createOrRotateCerts","object_name":"tempo-simplest-compactor-mtls","object_kind":"&TypeMeta{Kind:,APIVersion:,}","error":"secrets \"tempo-simplest-compactor-mtls\" is forbidden: unable to create new content in namespace chainsaw-loyal-drake because it is being terminated","stacktrace":"github.com/grafana/tempo-operator/internal/certrotation/handlers.CreateOrRotateCertificates\n\t/workspace/internal/certrotation/handlers/rotate_certs.go:78\ngithub.com/grafana/tempo-operator/controllers/tempo.(*TempoStackReconciler).Reconcile\n\t/workspace/controllers/tempo/tempostack_controller.go:125\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.3/pkg/internal/controller/controller.go:119\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.3/pkg/internal/controller/controller.go:316\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.3/pkg/internal/controller/controller.go:266\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.3/pkg/internal/controller/controller.go:227"}
{"level":"error","ts":"2024-08-29T09:41:47.133025777Z","logger":"tempostack-reconcile","msg":"failed to configure resource","controller":"tempostack","controllerGroup":"tempo.grafana.com","controllerKind":"TempoStack","TempoStack":{"name":"simplest","namespace":"chainsaw-loyal-drake"},"namespace":"chainsaw-loyal-drake","name":"simplest","reconcileID":"cd3b8daf-83ff-49c4-8acd-4ef8359e7803","tempostacks":"chainsaw-loyal-drake/simplest","event":"createOrRotateCerts","object_name":"tempo-simplest-gateway-mtls","object_kind":"&TypeMeta{Kind:,APIVersion:,}","error":"secrets \"tempo-simplest-gateway-mtls\" is forbidden: unable to create new content in namespace chainsaw-loyal-drake because it is being terminated","stacktrace":"github.com/grafana/tempo-operator/internal/certrotation/handlers.CreateOrRotateCertificates\n\t/workspace/internal/certrotation/handlers/rotate_certs.go:78\ngithub.com/grafana/tempo-operator/controllers/tempo.(*TempoStackReconciler).Reconcile\n\t/workspace/controllers/tempo/tempostack_controller.go:125\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.3/pkg/internal/controller/controller.go:119\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.3/pkg/internal/controller/controller.go:316\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.3/pkg/internal/controller/controller.go:266\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.3/pkg/internal/controller/controller.go:227"}
{"level":"error","ts":"2024-08-29T09:41:47.140857109Z","logger":"tempostack-reconcile","msg":"failed to configure resource","controller":"tempostack","controllerGroup":"tempo.grafana.com","controllerKind":"TempoStack","TempoStack":{"name":"simplest","namespace":"chainsaw-loyal-drake"},"namespace":"chainsaw-loyal-drake","name":"simplest","reconcileID":"cd3b8daf-83ff-49c4-8acd-4ef8359e7803","tempostacks":"chainsaw-loyal-drake/simplest","event":"createOrRotateCerts","object_name":"tempo-simplest-distributor-mtls","object_kind":"&TypeMeta{Kind:,APIVersion:,}","error":"secrets \"tempo-simplest-distributor-mtls\" is forbidden: unable to create new content in namespace chainsaw-loyal-drake because it is being terminated","stacktrace":"github.com/grafana/tempo-operator/internal/certrotation/handlers.CreateOrRotateCertificates\n\t/workspace/internal/certrotation/handlers/rotate_certs.go:78\ngithub.com/grafana/tempo-operator/controllers/tempo.(*TempoStackReconciler).Reconcile\n\t/workspace/controllers/tempo/tempostack_controller.go:125\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.3/pkg/internal/controller/controller.go:119\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.3/pkg/internal/controller/controller.go:316\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.3/pkg/internal/controller/controller.go:266\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.3/pkg/internal/controller/controller.go:227"}
error creating or rotate certificates failed to create or rotate TempoStack certificates

-->Reconciling tempostack chainsaw-loyal-drake/simplest
not found error TempoStack.tempo.grafana.com "simplest" not found
{"level":"error","ts":"2024-08-29T09:41:47.158544812Z","msg":"Reconciler error","controller":"tempostack","controllerGroup":"tempo.grafana.com","controllerKind":"TempoStack","TempoStack":{"name":"simplest","namespace":"chainsaw-loyal-drake"},"namespace":"chainsaw-loyal-drake","name":"simplest","reconcileID":"cd3b8daf-83ff-49c4-8acd-4ef8359e7803","error":"tempostacks.tempo.grafana.com \"simplest\" not found","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.3/pkg/internal/controller/controller.go:329\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.3/pkg/internal/controller/controller.go:266\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.3/pkg/internal/controller/controller.go:227"}

-->Reconciling tempostack chainsaw-loyal-drake/simplest
not found error TempoStack.tempo.grafana.com "simplest" not found

The alert TempoOperatorReconcileError is still firing, but that is expected.

      - alert: TempoOperatorReconcileError
        annotations:
          message: Tempo Operator failed to reconcile.
          runbook_url: https://github.com/grafana/tempo-operator/blob/main/operations/runbook.md#TempoOperatorReconcileError
        expr: |
          increase(controller_runtime_reconcile_errors_total{service="tempo-operator-controller-manager-metrics-service"}[15m]) > 0
          and
          increase(controller_runtime_terminal_reconcile_errors_total{service="tempo-operator-controller-manager-metrics-service"}[15m]) == 0
        for: 15m
        labels:
          severity: warning

Signed-off-by: Pavol Loffay <p.loffay@gmail.com>
@codecov-commenter
Copy link

Codecov Report

Attention: Patch coverage is 0% with 2 lines in your changes missing coverage. Please review.

Project coverage is 72.92%. Comparing base (bdf452d) to head (3df7293).

Files with missing lines Patch % Lines
controllers/tempo/tempomonolithic_controller.go 0.00% 1 Missing ⚠️
controllers/tempo/tempostack_controller.go 0.00% 1 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main    #1024      +/-   ##
==========================================
- Coverage   72.94%   72.92%   -0.03%     
==========================================
  Files         106      106              
  Lines        6575     6577       +2     
==========================================
  Hits         4796     4796              
- Misses       1489     1491       +2     
  Partials      290      290              
Flag Coverage Δ
unittests 72.92% <0.00%> (-0.03%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@IshwarKanse
Copy link
Contributor

@pavolloffay Verified the fix with the steps from the jira issue. We are good to merge the PR.

@pavolloffay pavolloffay merged commit e312c82 into grafana:main Aug 29, 2024
11 of 12 checks passed
@@ -82,6 +82,8 @@ func (r *TempoStackReconciler) Reconcile(ctx context.Context, req ctrl.Request)
log.Error(err, "unable to fetch TempoStack")
return ctrl.Result{}, fmt.Errorf("could not fetch tempo: %w", err)
}
// instance is not found, metrics can be cleared
status.ClearTempoStackMetrics(req.Namespace, req.Name)
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

do we always get a reconcile event on deletion? could we remove the ClearTempoStackMetrics() from the deletion webhook then?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants