certificate rotating

[yuriafy]

Is there an existing issue for this?

• I have searched the existing issues

Current Behavior

Greetings,
I am creating this issue to explain one issue I am facing.
I wanted to upgrade to one of the latest version of kubewarden on our test cluster :

• from crd version 1.6.0 to 1.10.0
• controller version 2.2.0 to 3.1.0
• policy-server from 1.16.0 to 1.18.0

I saw the removal of the cert-manager dependency starting from version 1.17 and I thought it was a good opportunity to upgrade. However I'm facing an issue with the certificate being rotate at some point, but the policy-server is still using the old one (or at least some component) causing the following error (in kube-api server logs):

W0102 08:43:37.127730 1 dispatcher.go:217] Failed calling webhook, failing closed clusterwide-cluster-container-resources-policy-audit.kubewarden.admission: failed calling webhook "clusterwide-cluster-container-resources-policy-audit.kubewarden.admission": failed to call webhook: Post "https://policy-server-default.kubewarden.svc:8443/validate/clusterwide-cluster-container-resources-policy-audit?timeout=30s": tls: failed to verify certificate: x509: certificate signed by unknown authority (possibly because of "x509: ECDSA verification failure" while trying to verify candidate authority certificate "kubewarden-controller-ca")

This is causing the whole cluster to reject any creation or update.
This is happening about every one or two days, and to solve the issue, I have to delete one (Cluster)AdmissionPolicy which is restarting the policy server and everything works again for some days.

It is also interesting to note that I am using ArgoCD to keep in sync every resources and no changes has been made by myself. When I disabled auto sync from argoCD, we can see that the certificate (in the secret) has been changed but since it's not syncing, the cluster is not applying this change and thus, everything is still working.

I have two questions about this issue, why is the certificate rotating even when it is still valid ? And also even if the certificate rotated why it seems some component is keeping the old version of it until it is restarted

Expected Behavior

No response

Steps To Reproduce

Using argoCD, pulling from

• repoURL: https://charts.kubewarden.io
  targetRevision: 3.1.0
  chart: kubewarden-controller

• repoURL: https://charts.kubewarden.io
  targetRevision: 1.10.0
  chart: kubewarden-crds

Environment

- OS: Linux
- Architecture: x86_64

Anything else?

Log from controller

{"level":"info","ts":"2024-12-30T14:04:59Z","logger":"controller-runtime.certwatcher","msg":"Updated current TLS certificate"}
{"level":"info","ts":"2024-12-30T14:04:59Z","logger":"controller-runtime.certwatcher","msg":"Updated current TLS certificate"}
{"level":"info","ts":"2024-12-30T14:04:59Z","logger":"controller-runtime.certwatcher","msg":"Updated current TLS certificate"}
{"level":"info","ts":"2024-12-30T14:04:59Z","logger":"controller-runtime.certwatcher","msg":"Updated current TLS certificate"}
{"level":"info","ts":"2024-12-30T21:35:03Z","logger":"cert-recociler","msg":"Certificate verification failed, rotating the certificate","dnsName":"policy-server-default.kubewarden.svc","verification error":"the certificate is invalid: x509: certificate signed by unknown authority (possibly because of \"x509: ECDSA verification failure\" while trying to verify candidate authority certificate \"kubewarden-controller-ca\")"}
{"level":"info","ts":"2024-12-30T21:35:03Z","logger":"cert-recociler","msg":"Certificate rotated successfully","dnsName":"policy-server-default.kubewarden.svc"}
{"level":"info","ts":"2024-12-31T14:05:01Z","logger":"controller-runtime.certwatcher","msg":"Updated current TLS certificate"}
{"level":"info","ts":"2024-12-31T14:05:01Z","logger":"controller-runtime.certwatcher","msg":"Updated current TLS certificate"}
{"level":"info","ts":"2024-12-31T14:05:01Z","logger":"controller-runtime.certwatcher","msg":"Updated current TLS certificate"}
{"level":"info","ts":"2024-12-31T14:05:01Z","logger":"controller-runtime.certwatcher","msg":"Updated current TLS certificate"}
{"level":"info","ts":"2024-12-31T14:35:19Z","logger":"controller-runtime.certwatcher","msg":"Updated current TLS certificate"}
{"level":"info","ts":"2024-12-31T14:35:19Z","logger":"controller-runtime.certwatcher","msg":"Updated current TLS certificate"}
{"level":"info","ts":"2024-12-31T14:35:19Z","logger":"controller-runtime.certwatcher","msg":"Updated current TLS certificate"}
{"level":"info","ts":"2024-12-31T14:35:19Z","logger":"controller-runtime.certwatcher","msg":"Updated current TLS certificate"}
{"level":"info","ts":"2024-12-31T21:35:03Z","logger":"cert-recociler","msg":"Certificate verification failed, rotating the certificate","dnsName":"policy-server-default.kubewarden.svc","verification error":"the certificate is invalid: x509: certificate signed by unknown authority (possibly because of \"x509: ECDSA verification failure\" while trying to verify candidate authority certificate \"kubewarden-controller-ca\")"}
{"level":"info","ts":"2024-12-31T21:35:03Z","logger":"cert-recociler","msg":"Certificate rotated successfully","dnsName":"policy-server-default.kubewarden.svc"}
{"level":"info","ts":"2025-01-01T14:38:34Z","logger":"controller-runtime.certwatcher","msg":"Updated current TLS certificate"}
{"level":"info","ts":"2025-01-01T14:38:34Z","logger":"controller-runtime.certwatcher","msg":"Updated current TLS certificate"}
{"level":"info","ts":"2025-01-01T14:38:34Z","logger":"controller-runtime.certwatcher","msg":"Updated current TLS certificate"}
{"level":"info","ts":"2025-01-01T14:38:34Z","logger":"controller-runtime.certwatcher","msg":"Updated current TLS certificate"}
{"level":"info","ts":"2025-01-01T21:35:03Z","logger":"cert-recociler","msg":"Certificate verification failed, rotating the certificate","dnsName":"policy-server-default.kubewarden.svc","verification error":"the certificate is invalid: x509: certificate signed by unknown authority (possibly because of \"x509: ECDSA verification failure\" while trying to verify candidate authority certificate \"kubewarden-controller-ca\")"}
{"level":"info","ts":"2025-01-01T21:35:03Z","logger":"cert-recociler","msg":"Certificate rotated successfully","dnsName":"policy-server-default.kubewarden.svc"}

Secret before update

secret: kubewarden-ca
ca.crt
-----BEGIN CERTIFICATE-----
MIIBpTCCAUygAwIBAgIRANfZFRczATR72wxnN5BUWWwwCgYIKoZIzj0EAwIwIzEh
MB8GA1UEAxMYa3ViZXdhcmRlbi1jb250cm9sbGVyLWNhMB4XDTI0MTIzMDA5MzI1
N1oXDTM0MTIyODA5MzI1N1owIzEhMB8GA1UEAxMYa3ViZXdhcmRlbi1jb250cm9s
bGVyLWNhMFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAEWA+bhMq6wuZ4T8b3Z3aF
aT4joMRxoNGZwbg6E4MvUQCQhRbAHgacLEys3HjME9S7S0xLhEtnTs49NRQdJdy+
EKNhMF8wDgYDVR0PAQH/BAQDAgKkMB0GA1UdJQQWMBQGCCsGAQUFBwMBBggrBgEF
BQcDAjAPBgNVHRMBAf8EBTADAQH/MB0GA1UdDgQWBBQc8P9CPoMGTDSgIR54JhJM
J0AdyDAKBggqhkjOPQQDAgNHADBEAiBsCV1e/qzFUZPwAYl7OC3urGkPGetbxf6Q
YysNM8nYCwIgbnOQi9Z74JBclJo+89Gh6Jt/NTwY1rz0jMSJrLHCRaQ=
-----END CERTIFICATE-----
ca.key
-----BEGIN EC PRIVATE KEY-----
MHcCAQEEIDislXpOeOjOJayoYq4wveE88Q5B39QdqZ6pK8NZBaJtoAoGCCqGSM49
AwEHoUQDQgAEWA+bhMq6wuZ4T8b3Z3aFaT4joMRxoNGZwbg6E4MvUQCQhRbAHgac
LEys3HjME9S7S0xLhEtnTs49NRQdJdy+EA==
-----END EC PRIVATE KEY-----

secret: policy-reporter-default
tls.crt
-----BEGIN CERTIFICATE-----
MIIB1DCCAXugAwIBAgIQNyid8GWImchpY/mlVzaHSDAKBggqhkjOPQQDAjAjMSEw
HwYDVQQDExhrdWJld2FyZGVuLWNvbnRyb2xsZXItY2EwHhcNMjQxMjMwMDkzNTA3
WhcNMjUxMjMwMDkzNTA3WjAvMS0wKwYDVQQDEyRwb2xpY3ktc2VydmVyLWRlZmF1
bHQua3ViZXdhcmRlbi5zdmMwWTATBgcqhkjOPQIBBggqhkjOPQMBBwNCAAQ9N0GC
jrl5vVmLIUPJhJu2709KZwzdTS7IsOGfW4qPIMdup7J/nkr1F2qc4crxmC7ljanX
pJvp1dWpvARnb4SWo4GEMIGBMA4GA1UdDwEB/wQEAwIFoDAdBgNVHSUEFjAUBggr
BgEFBQcDAgYIKwYBBQUHAwEwHwYDVR0jBBgwFoAUHPD/Qj6DBkw0oCEeeCYSTCdA
HcgwLwYDVR0RBCgwJoIkcG9saWN5LXNlcnZlci1kZWZhdWx0Lmt1YmV3YXJkZW4u
c3ZjMAoGCCqGSM49BAMCA0cAMEQCIHzcazPGaQCqtMWRIutdOY9CltH0uO+nJgZE
SORmR8ztAiB8EwMyIXoW0+ZIx+3SdzPtikN+146lsvTShgeH0Cz/Ig==
-----END CERTIFICATE-----

kubewarden-webhook-server-cert
tls.crt
-----BEGIN CERTIFICATE-----
MIICAzCCAamgAwIBAgIQC9kGQqRL2oO3D44HunB2iDAKBggqhkjOPQQDAjAjMSEw
HwYDVQQDExhrdWJld2FyZGVuLWNvbnRyb2xsZXItY2EwHhcNMjQxMjMwMDkzMjU3
WhcNMjUxMjMwMDkzMjU3WjA/MT0wOwYDVQQDEzRrdWJld2FyZGVuLWNvbnRyb2xs
ZXItd2ViaG9vay1zZXJ2aWNlLmt1YmV3YXJkZW4uc3ZjMFkwEwYHKoZIzj0CAQYI
KoZIzj0DAQcDQgAERRH6d7jS32BPoba/xYT2OgZ0PAabjDj7mZM0lZNiGXwImyDK
XHKLH+yBf3D3BFEfyGmQDkfG46zyRsGL/7U1LKOBojCBnzAOBgNVHQ8BAf8EBAMC
BaAwHQYDVR0lBBYwFAYIKwYBBQUHAwEGCCsGAQUFBwMCMAwGA1UdEwEB/wQCMAAw
HwYDVR0jBBgwFoAUHPD/Qj6DBkw0oCEeeCYSTCdAHcgwPwYDVR0RBDgwNoI0a3Vi
ZXdhcmRlbi1jb250cm9sbGVyLXdlYmhvb2stc2VydmljZS5rdWJld2FyZGVuLnN2
YzAKBggqhkjOPQQDAgNIADBFAiEA2uE4tHEiZvhDFXq99pQTGQzacRKNNquE5k1d
mAaoGvQCIFUNLXDtzFQKjMkYoxXmmqJ/5rDEgx9VqRh4k/Zh671M
-----END CERTIFICATE-----

Secret after update

secret: kubewarden-ca
ca.crt
-----BEGIN CERTIFICATE-----
MIIBpTCCAUugAwIBAgIQEoPB95ptpq+NCZG82u6gFTAKBggqhkjOPQQDAjAjMSEw
HwYDVQQDExhrdWJld2FyZGVuLWNvbnRyb2xsZXItY2EwHhcNMjUwMTAxMTQzNzUz
WhcNMzQxMjMwMTQzNzUzWjAjMSEwHwYDVQQDExhrdWJld2FyZGVuLWNvbnRyb2xs
ZXItY2EwWTATBgcqhkjOPQIBBggqhkjOPQMBBwNCAARrUGHJ2OMNY7bgWPccYLtV
f672ntmaJv3OgsMaypz4C6UMm986kUTQUMfb9L8WTs2OMs8lYkhPLCcvdRoGJlFm
o2EwXzAOBgNVHQ8BAf8EBAMCAqQwHQYDVR0lBBYwFAYIKwYBBQUHAwEGCCsGAQUF
BwMCMA8GA1UdEwEB/wQFMAMBAf8wHQYDVR0OBBYEFCtYFMOfNlf9bJzPYOof9M9g
he5uMAoGCCqGSM49BAMCA0gAMEUCIATX4mCzT+4e0SzMqp8cOP1kmfnkGdz4f8t6
U/QFo4pPAiEAsXh+Db5w35zdiGotuiyO/HgrD80GPaZZ9SpappbDZKs=
-----END CERTIFICATE-----

secret: policy-reporter-default
tls.crt
-----BEGIN CERTIFICATE-----
MIIB1TCCAXugAwIBAgIQPnQpQLw1wJODGAGjUCAC6TAKBggqhkjOPQQDAjAjMSEw
HwYDVQQDExhrdWJld2FyZGVuLWNvbnRyb2xsZXItY2EwHhcNMjUwMTAxMjEzNTAz
WhcNMjYwMTAxMjEzNTAzWjAvMS0wKwYDVQQDEyRwb2xpY3ktc2VydmVyLWRlZmF1
bHQua3ViZXdhcmRlbi5zdmMwWTATBgcqhkjOPQIBBggqhkjOPQMBBwNCAATg/lcj
QQki0B9TN4+A+vTybBI8GQa3nVBVjAIwExNTgye1ksCiWg1B/7z/VnxfNkVf74id
62RtumNYsoVaNh74o4GEMIGBMA4GA1UdDwEB/wQEAwIFoDAdBgNVHSUEFjAUBggr
BgEFBQcDAgYIKwYBBQUHAwEwHwYDVR0jBBgwFoAUK1gUw582V/1snM9g6h/0z2CF
7m4wLwYDVR0RBCgwJoIkcG9saWN5LXNlcnZlci1kZWZhdWx0Lmt1YmV3YXJkZW4u
c3ZjMAoGCCqGSM49BAMCA0gAMEUCIAyjI8JjjbnZAu8bnxJpLDpwA00J8aNjGErV
qwl1XcB1AiEAyJKgyLraNYax4dQbvCulrv7pKILuju6ZYWEkQ8143Xg=
-----END CERTIFICATE-----

kubewarden-webhook-server-cert
tls.crt
-----BEGIN CERTIFICATE-----
MIICAzCCAamgAwIBAgIQHgW5vnKOD7BfGPm57lp0DTAKBggqhkjOPQQDAjAjMSEw
HwYDVQQDExhrdWJld2FyZGVuLWNvbnRyb2xsZXItY2EwHhcNMjUwMTAxMTQzNzUz
WhcNMjYwMTAxMTQzNzUzWjA/MT0wOwYDVQQDEzRrdWJld2FyZGVuLWNvbnRyb2xs
ZXItd2ViaG9vay1zZXJ2aWNlLmt1YmV3YXJkZW4uc3ZjMFkwEwYHKoZIzj0CAQYI
KoZIzj0DAQcDQgAErcpOkkGNTMCPnXtTXSulx2d3D2A2iuNQ+814kXetv4BBpL8p
cnbK4l0kbeiXyDbEg/zjcCflACC1bBo+jiGTvaOBojCBnzAOBgNVHQ8BAf8EBAMC
BaAwHQYDVR0lBBYwFAYIKwYBBQUHAwEGCCsGAQUFBwMCMAwGA1UdEwEB/wQCMAAw
HwYDVR0jBBgwFoAUK1gUw582V/1snM9g6h/0z2CF7m4wPwYDVR0RBDgwNoI0a3Vi
ZXdhcmRlbi1jb250cm9sbGVyLXdlYmhvb2stc2VydmljZS5rdWJld2FyZGVuLnN2
YzAKBggqhkjOPQQDAgNIADBFAiEA9lOmyen2ADkTUtzrLc2S+++rXfKDvrvD3+dv
3Fs17U4CIEgud6Sx4Xx6y/6jYjAu5sdI8gcVlIQgWFpu9vTsku2o
-----END CERTIFICATE-----

[viccuad]

Hi,

Could you clarify the Helm chart versions of kubewarden-crds, kubewarden-controller, kubewarden-defaults prior to the upgrade and after the upgrade?

I'm trying to find the matching versions of the images and helm charts that match what youy shared to narrow down the problem, yet I think you have made a typo (e.g: crds version 1.8.0 instead of 1.6.0?).

Just in case, the charts must be upgraded in lockstep with their appVersion.

Also, in rare cases, one needs to follow the upgrade path of upgrade until latest patch, then minor version (see here). I don't think this is the case here, but just in case.

[jvanz]

Let me try to simulate this...