Resources dispatched to namespaces are not removed when relations removed

[misohu]

It is expected that, when a relation asking for a resource to be injected into user namespaces (say a Secret) is removed, that the resource should be removed from all user namespaces. At present, resource-dispatcher fails to remove these resources generated by metacontroller when the relation creating them is removed.

Way to reproduce:

• Deploy metacontroller charm

juju deploy metacontroller-operator --trust

• Build and deploy charm

charmcraft pack
juju deploy ./*.charm --resource oci-image=$(yq eval '.resources.oci-image.upstream-source' metadata.yaml) --trust

• Deploy at least two members of the relation (you can use the test charm from integration tests)

cd tests/integration/manifests-tester
charmcraft pack
juju deploy ./*.charm --trust
juju deploy ./*.charm manifests-tester2 --config test_data=src/secrets2

• Create relations

juju relate manifests-tester:secrets resource-dispatcher:secrets
juju relate manifests-tester2:secrets resource-dispatcher:secrets

• Create example namespace

kubectl apply -f - <<EOF
apiVersion: v1
kind: Namespace
metadata:
  name: test-namespace
  labels:
    user.kubeflow.org/enabled: "true"
EOF

Now you can check that secrets are created for both of the manifests-testers.

kubectl get secrets -n test-namespace

Now remove one relation.

juju remove-relation resource-dispatcher manifests-tester

Although the relation is removed and dispatcher is clearly not sending the old secrets, metacontroller wont remove the extra secrets. You can check.

kubectl get secrets -n test-namespace

After checking metacontroller logs

kubectl logs -n kubeflow metacontroller-operator-charm-0 > logs.log

We are getting

Failed to sync Namespace "misko-test2": can't reconcile children for Namespace /misko-test2: [can't delete Secret misko-test2/mlpipeline-minio-artifact: the server could not find the requested resource, can't delete Secret misko-test2/seldon-rclone-secret: the server could not find the requested resource]