containerd is not able to authenticate and pull image from GCP Artifact Registry

[alex-ioma]

Summary

crictl or ctr is not able to pull images from Artifact Registry via Service Account Key. The service account can correctly pull images if the related key is added as .dockerconfig secret within k8s but it does not work if added as a docker config file to the underlying containerd.

crictl version v1.26.1
ctr github.com/k3s-io/containerd v1.7.11-k3s2

What Should Happen Instead?

Images are pulled.

Reproduction Steps

1. Add service account key json-formatted file as containerd mirror auth

e.g. via config.toml.tmpl

[plugins.cri.registry]
  [plugins.cri.registry.mirrors]
    [plugins.cri.registry.mirrors."europe-docker.pkg.dev"]
      endpoint = ["https://europe-docker.pkg.dev"]
[plugins.cri.registry.auths]
  [plugins.cri.registry.auths."https://europe-docker.pkg.dev"]
    auth = "base64-encoded service account key file"

3. Try and pull image
4. Get the following error

E0606 10:48:25.860886 3716290 remote_image.go:171] "PullImage from image service failed" err="rpc error: code = Unknown desc = failed to pull and unpack image \"europe-docker.pkg.dev/repo/image:v0.1.0-alpha\": failed to resolve reference \"europe-docker.pkg.dev/repo/image:v0.1.0-alpha\": failed to authorize: failed to fetch oauth token: unexpected status from GET request to https://europe-docker.pkg.dev/v2/token?scope=repository%3Acm-controls%2Flgm%2Flgm-hmi%3Apull&service=europe-docker.pkg.dev: 401 Unauthorized" image="europe-docker.pkg.dev/repo/image:v0.1.0-alpha"
FATA[0000] pulling image: rpc error: code = Unknown desc = failed to pull and unpack image "europe-docker.pkg.dev/repo/image:v0.1.0-alpha": failed to resolve reference "europe-docker.pkg.dev/repo/image:v0.1.0-alpha": failed to authorize: failed to fetch oauth token: unexpected status from GET request to https://europe-docker.pkg.dev/v2/token?scope=repository%3Acm-controls%2Flgm%2Flgm-hmi%3Apull&service=europe-docker.pkg.dev: 401 Unauthorized

Introspection Report

I believe the issue is on how the auth is parsed and used. The key is a long-lived json-key file that should use an RSA key to retrieve a oauth JWT token via Bearer token from the google token_uri. It looks like containerd is trying to get the token from the repository URI

Can you suggest a fix?

Not sure how to troubleshoot further but might be similar to issue #990

Are you interested in contributing with a fix?

Happy to test possible solution strategies.

[kakkoyun]

I'm having the same issue. It's not clear anymore from the documentation (https://github.com/containerd/containerd/blob/main/docs/cri/registry.md and https://github.com/containerd/containerd/blob/main/docs/cri/config.md#registry-configuration) how to add credentials for private registries. The documents mention deprecated configuration surfaces and even mention there won't be a way to configure authentication with plain text. I tried to read the code, but it's a rabbit hole.

What's the best way to configure private registry access for CRI?

us-central1-docker.pkg.dev

[neoaggelos]

Hi folks, I believe the right way to configure this would be https://github.com/containerd/cri/blob/master/docs/registry.md#configure-registry-credentials-example---gcr-with-service-account-key-authentication ?

[alex-ioma]

Hi folks, I believe the right way to configure this would be https://github.com/containerd/cri/blob/master/docs/registry.md#configure-registry-credentials-example---gcr-with-service-account-key-authentication ?

Thanks @neoaggelos but that is not ok.

The manual use of Service Account Key to generate a Token (and then use that token in the docker auth) is not a sustainable approach. This is because that token is a short-lived token that needs to be regenerated with a refresh-token via a oath approach. Such approach must be handled directly by the containerd process by using the Service Account Key (the key needs to be generated with the least possible permission on GPC IAM section, of course).

To be clear, Kubernetes is already able to manage such oauth token/refresh-token approach by providing GCP key as auth parameter of the .dockerconfig secrets. But such special secret needs to be added to each namespace that needs to pull images from that registry. However, the same approach doesn't work if someone wants to implemented it at containerd "raw" level.

Different post online reports possible bug / issues with conatinerd itself that is not able to manage such oauth process but I'm not sure.

I was hoping to have at least a confirmation from the canonical team.

[neoaggelos]

Hi @alex-ioma, thank you for elaborating on why this is not a sustainable approach.

Overall, indeed, this looks something containerd specific, and not a limitation or bug on the side of MicroK8s. I believe raising an issue at https://github.com/containerd/containerd might be more useful towards a resolution.

[BrennenMM7]

@alex-ioma 👋 Hello random passerby here. When it comes to authenticating to the the Artifact Registry service offered by GCP for pulling container images, Google uses their own binary for performing the token auth automatically for GKE clusters.

They define a CredentialProviderConfig on the kubelet in addition pass args like image-credential-provider-config that use this binary from cloud-provider repo.

I'm not using microk8s, building my own image stuff for cluster-api work, but I thought this might help you in a better direction if you can get your kubelet to perform the same operation?

Update: Confirmed this does work on my cluster 💯

Example:

apiVersion: kubelet.config.k8s.io/v1
providers:
  - name: auth-provider-gcp
    apiVersion: credentialprovider.kubelet.k8s.io/v1
    matchImages:
    - "container.cloud.google.com"
    - "gcr.io"
    - "*.gcr.io"
    - "*.pkg.dev"
    args:
    - get-credentials
    - --v=3
    defaultCacheDuration: 1m```