Add option for imagePullSecrets in podOptions

[sandert-k8s]

Describe the feature

Add imagePullSecrets to the tenant CR, .spec.podOptions.imagePullSecrets. These imagePullSecrets gets automatically appended to the pod spec when a pod in the tenant is deployed. This is great when a GlobalTenantResource with imagePullSecrets is used, so these imagePullSecrets are automatically added and the tenantUser/owner doesn't have to do a thing for it :)

What would the new user story look like?

How would the new interaction with Capsule look like? E.g.

1. Tenant owner creates a new Pod
2. The imagePullSecrets are added to the Pod
3. The tenant owner is happy and worryfree about these imagePullSecrets that we manage for him and he doesn't have to think about adding it to his podSpec.

apiVersion: capsule.clastix.io/v1beta2
kind: Tenant
metadata:
  name: tenant-a
  labels:
    tenant: a
spec:
  podOptions:
    imagePullSecrets:
        - name: secret-proxy-docker
        - name: secret-proxy-ghcr
  contianerRegistries:
    allowed:
      - thisismy.containerregistry.io

And just for reference, to get the global picture, the globaltenantresource:

apiVersion: capsule.clastix.io/v1beta2
kind: GlobalTenantResource
metadata:
  name: pullsecrets-tenant-a
spec:
  tenantSelector:
    matchLabels:
      tenant: a
  resyncPeriod: 60s
  resources:
    - namespacedItems:
        - apiVersion: v1
          kind: Secret
          namespace: tenant-a-secrets
          selector:
            matchLabels:
              imagePullSecret: thisismy.containerregistry.io

Expected behavior

ImagePullSecrets are automatically added to all pods in a tenant.

[oliverbaehler]

Same here as with the annotations, we will add a new Hook system, Kyverno like, to apply such patches based on selected tenants or namespaces, decoupling it from tenants

[sandert-k8s]

Yup that sounds good to me! 👍

[prometherion]

In the meanwhile, since you're using Tenant Resources, you could update the default ServiceAccount on each Namespace where the secret is replicated to use the provided image pull secret.

https://kubernetes.io/docs/tasks/configure-pod-container/configure-service-account/#add-image-pull-secret-to-service-account