feat: containerd configuration for mirror registry

[supershal]

Closes: #203

✔️ Create containerd mirror configuration for a registry
❌ Create Dynamic Credentials Provider configuration to use static mirror credentials
❌ Update Dynamic Credentials Provider configuration to use mirror credentials only with MirrorCredentialOnly strategy
✔️ Unit tests to check the generated kubeadm files
✔️ Updated documentation for global mirror

    variables:
      - name: clusterConfig
        value:
          imageRegistries:
            - url: https://my-registry.io
              credentials:
                secretRef:
                  name: my-registry-credentials
          globalImageRegistryMirror:
            - url: https://my-mirror.io
              credentials:
                secretRef:
                  name: my-mirror-CA-credentials

Future PRs:
Following Changes were making the current PR long and complicate to follow. I think we can integrate with konvoy using current changes in the PR.

• Create Dynamic Credentials Provider configuration to use static mirror credentials
• Update Dynamic Credentials Provider configuration to use mirror credentials only with MirrorCredentialOnly strategy

[supershal]

Still running manual test with ECR registry mirror. Once the test completed, I will turn the draft PR to Ready.

[dkoshkin]

Thank you for all the tests!

[supershal]

Tested that mirror configuration is created on AWS VMS.
steps:

1. Create ECR registry and push kubernetes images

./dkp push bundle \
--bundle kubernetes-images-1.27.6-d2iq.1.tar \
--to-registry 999867407951.dkr.ecr.us-west-2.amazonaws.com/shalin-mirror \
--to-registry-username AWS \
--to-registry-password $(aws ecr get-login-password) \
--ecr-lifecycle-policy-file ecr-lifecycle-policy.json

2. create aws creds secrets as per README file
3. Create cluster using mirror variable

apiVersion: cluster.x-k8s.io/v1beta1
kind: Cluster
metadata:
  labels:
    cluster.x-k8s.io/provider: aws
  name: aws-quick-start
spec:
  clusterNetwork:
    pods:
      cidrBlocks:
        - 192.168.0.0/16
    serviceDomain: cluster.local
    services:
      cidrBlocks:
        - 10.128.0.0/12
  topology:
    class: aws-quick-start
    controlPlane:
      replicas: 1
    variables:
      - name: clusterConfig
        value:
          addons:
            cni:
              provider: calico
            cpi: {}
            csi:
              providers:
                - name: aws-ebs
            nfd: {}
          aws:
            region: us-west-2
          imageRegistries:
            - url: https://999867407951.dkr.ecr.us-west-2.amazonaws.com/shalin-mirror
              mirror: {}
    version: v1.27.5
    workers:
      machineDeployments:
        - class: default-worker
          name: md-0
          replicas: 1
---
apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
kind: AWSClusterStaticIdentity
metadata:
  labels:
    cluster.x-k8s.io/provider: aws
  name: aws-quick-start
spec:
  allowedNamespaces:
    list:
      - default
  secretRef: aws-quick-start-creds

4. Checked that the kubeadmcontrolplane and kubeadmconfigtempnlate has following files

- content: |
    apiVersion: credentialprovider.d2iq.com/v1alpha1
    kind: DynamicCredentialProviderConfig
    mirror:
      endpoint: "999867407951.dkr.ecr.us-west-2.amazonaws.com/shalin-mirror"
      credentialsStrategy: "MirrorCredentialsOnly"
    credentialProviderPluginBinDir: /etc/kubernetes/image-credential-provider/
    credentialProviders:
      apiVersion: kubelet.config.k8s.io/v1beta1
      kind: CredentialProviderConfig
      providers:
      - name: ecr-credential-provider
        args:
        - get-credentials
        matchImages:
        - "999867407951.dkr.ecr.us-west-2.amazonaws.com/shalin-mirror"
        defaultCacheDuration: "0s"
        apiVersion: credentialprovider.kubelet.k8s.io/v1alpha1
  path: /etc/kubernetes/dynamic-credential-provider-config.yaml
  permissions: "0600"
- content: |
    [host."https://999867407951.dkr.ecr.us-west-2.amazonaws.com/shalin-mirror"]
      capabilities = ["pull", "resolve"]
  path: /etc/containerd/certs.d/_default/hosts.toml
  permissions: "0600"

5. Checked on the VMs in AWS that above files are created

The cluster partially came up, CNI pods were not starting.
However I am unable to determine from logs of kubelet or containerd if mirror was indeed used to pull image and mirror credentials were used by dynamic credentials provider. I will sync up with the team and figure out how to confirm if images pulled from the mirror in networked environment.

Another alternative is to create cluster in airgap environment with ECR mirror. however it will require a lot more additional setup manually which we are planning to perform in konvoy e2e test for cluser class anyways.

[jimmidyson]

What happens if multiple clusters are marked as mirror? I wonder if the config should look more like:

variables:
  - name: clusterConfig
    value:
      imageRegistries:
        credentials:
          - url: https://my-registry.io/
            secretRef:
              name: my-registry-credentials
        mirror: https://my-registry.io/

[jimmidyson]

Please review and merge #293 and then rebase this PR.

[supershal]

What happens if multiple clusters are marked as mirror? I wonder if the config should look more like:

variables:
  - name: clusterConfig
    value:
      imageRegistries:
        credentials:
          - url: https://my-registry.io/
            secretRef:
              name: my-registry-credentials
        mirror: https://my-registry.io/

We did considered this during our discussions.
konvoy currently supports only one mirror registry and credentials for that mirror registry. This can change in future.
Current implementation errors out if more than one registry is provided.

IMHO imageRegistries should be an array of imageRegistry with each registry can have one set of static credentials and each potentially can be used as a mirror.

variables:
   - name: clusterConfig
     value:
       imageRegistries:
           - url: https://registry-1.io/
              credentials:           
                  secretRef:
                     name: registry-1-credentials
              mirror: {}
           - url: https://registry-2.io/
              credentials:           
                  secretRef:
                     name: registry-2-credentials
              mirror: 
                 secretRef:
                     name: registry-2-CA-secret

We have a google doc with all suggested options for the schema. I will start a thread in slack to discuss it.

[supershal]

Created stacked PR #296 to fix flakey unit tests. blocking this PR until its merged.

Schema changed a lot. please review again.

[jimmidyson]

Pushed changes from comments, thanks for this PR @supershal!

[supershal]

Thank you @jimmidyson for the final changes.