secrets are deleted when delete-psmdb-pvc has been set

[m-bahmann]

percona-server-mongodb-operator/pkg/controller/perconaservermongodb/finalizers.go

Line 204 in 7b1610a

err = r.deleteSecrets(ctx, cr)

secrets are deleted when delete-psmdb-pvc has been set.
the problem is sometimes the percona secrets (cr.Spec.Secrets.Users,
"internal-" + cr.Name + "-users") are respectively being recreated right after the deletion in reconcileUsersSecret, reconcileUsers func calls. Sometimes not.

This logic of deleting secrets causes issues/side effects in tha case that the psmdb CR might be re-created after to recover a deleted cluster for example.

To work around this logic we try to overwrite/create the user secret with the previous data (leaving to the operator to sync the secrets, etc), but sometimes recreating psmdb lateron ends up creating the mongodb pods successfully with some auth errors on the mongo pods though, sometimes not. the psmdb does not go in initializing status even. (there are logs on the operator: not found the internal users secret.)

[spron-in]

Hey @m-bahmann ,

I'm not sure which issue you are after here.
So deletion of Secrets is desired behavior if delete-psmdb-pvc is set. We might add a separate finalizer for secrets. Is it what you are after?

Recreating PSMDB cluster without secrets on new PVC should not be the problem.

Also I'm a bit confused with
the problem is sometimes the percona secrets (cr.Spec.Secrets.Users,
"internal-" + cr.Name + "-users") are respectively being recreated right after the deletion in reconcileUsersSecret, reconcileUsers func calls. Sometimes not.

What does it mean "Sometimes not"? Is it that there is an issue that secrets are not deleted sometimes?

[lkadalski]

Hi :)
I have the same issue. When using terraform and helm_release after destroy plan, there is still one secret internal-{name}-users remaining. This secret then is taken as preffered one for new cluster in that namespace. Regardless what we set in values.yaml for secrets.users. The issue is that it may contain outdated credentials. It would be nice to have an additional finalizer or any other solution/flag for cleaning up that secret.
Currently I have to delete secrets manually after each cluster destroy.
I have this issue specifically when using only one finalizer (delete-pvc).
When using both it usually works as expected.

[m-bahmann]

Hi, Thanks for looking into this. It would be a great idea to add a separate finalizer for the secrets. Currently for our use case we had to implement a workaround to not lose the secrets when deleting psmdb CR with the pvc finalizer.

…

On Wednesday, January 24, 2024, Sergey Pronin ***@***.***> wrote: Hey @m-bahmann <https://github.com/m-bahmann> , I'm not sure which issue you are after here. So deletion of Secrets is desired behavior if delete-psmdb-pvc is set. We might add a separate finalizer for secrets. Is it what you are after? Recreating PSMDB cluster without secrets on new PVC should not be the problem. Also I'm a bit confused with the problem is sometimes the percona secrets (cr.Spec.Secrets.Users, "internal-" + cr.Name + "-users") are respectively being recreated right after the deletion in reconcileUsersSecret, reconcileUsers func calls. Sometimes not. What does it mean "Sometimes not"? Is it that there is an issue that secrets are not deleted sometimes? — Reply to this email directly, view it on GitHub <#1409 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AZ2JOKNK2IQU5GWNLROEX3TYQFBFHAVCNFSM6AAAAABBL7CMUGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSMBYGYYTIOJZHE> . You are receiving this because you were mentioned.Message ID: ***@***.*** com>

[tplavcic]

@m-bahmann I'm interested in the failure when the secret is recreated after cluster deletion (I have tried to reproduce it with sharding but couldn't).
Could you share your cr.yaml or if not at least which config was used (sharding/replica, anything else interesting). I would like to try to reproduce it, but also if you have some steps to reproduce you can open a Jira ticket in K8SPSMDB project here: https://perconadev.atlassian.net/
Thanks!