containerd: preloaded images fail to save

[nirs]

What Happened?

Saving preloaded images (e.g. registry.k8s.io/pause:3.10) with containerd runtime is broken, creating empty tar. Saving images pulled into minikube works.

I suspect that the preloaded images were not prepared with contained, and their internal structure is not the same as the image pulled into the cluster by containerd.

How to reproduce:

macOS/vkfit

% minikube start --driver vfkit --container-runtime containerd
😄  minikube v1.36.0 on Darwin 15.6.1 (arm64)
✨  Using the vfkit driver based on user configuration
👍  Starting "minikube" primary control-plane node in "minikube" cluster
💾  Downloading Kubernetes v1.33.2 preload ...
    > preloaded-images-k8s-v18-v1...:  376.62 MiB / 376.62 MiB  100.00% 18.46 M
🔥  Creating vfkit VM (CPUs=2, Memory=6144MB, Disk=20000MB) ...
📦  Preparing Kubernetes v1.33.2 on containerd 1.7.23 ...
🔗  Configuring bridge CNI (Container Networking Interface) ...
🔎  Verifying Kubernetes components...
    ▪ Using image gcr.io/k8s-minikube/storage-provisioner:v5
🌟  Enabled addons: default-storageclass, storage-provisioner
🏄  Done! kubectl is now configured to use "minikube" cluster and "default" namespace by default


% minikube image ls
registry.k8s.io/pause:3.10
registry.k8s.io/kube-scheduler:v1.33.2
registry.k8s.io/kube-proxy:v1.33.2
registry.k8s.io/kube-controller-manager:v1.33.2
registry.k8s.io/kube-apiserver:v1.33.2
registry.k8s.io/etcd:3.5.21-0
registry.k8s.io/coredns/coredns:v1.12.0
gcr.io/k8s-minikube/storage-provisioner:v5
docker.io/kindest/kindnetd:v20250512-df8de77b

% minikube image save registry.k8s.io/pause:3.10 test.tar; echo $?
0

% file test.tar 
test.tar: empty

Linux/kvm

$ minikube start --driver kvm --network default --container-runtime containerd
😄  minikube v1.36.0 on Fedora 42 (kvm/amd64)
✨  Using the kvm2 driver based on user configuration
👍  Starting "minikube" primary control-plane node in "minikube" cluster
💾  Downloading Kubernetes v1.33.2 preload ...
    > preloaded-images-k8s-v18-v1...:  401.08 MiB / 401.08 MiB  100.00% 15.15 M
🔥  Creating kvm2 VM (CPUs=2, Memory=6144MB, Disk=20000MB) ...
📦  Preparing Kubernetes v1.33.2 on containerd 1.7.23 ...
🔗  Configuring bridge CNI (Container Networking Interface) ...
🔎  Verifying Kubernetes components...
    ▪ Using image gcr.io/k8s-minikube/storage-provisioner:v5
🌟  Enabled addons: storage-provisioner, default-storageclass
🏄  Done! kubectl is now configured to use "minikube" cluster and "default" namespace by default

$ minikube image ls
registry.k8s.io/pause:3.10
registry.k8s.io/kube-scheduler:v1.33.2
registry.k8s.io/kube-proxy:v1.33.2
registry.k8s.io/kube-controller-manager:v1.33.2
registry.k8s.io/kube-apiserver:v1.33.2
registry.k8s.io/etcd:3.5.21-0
registry.k8s.io/coredns/coredns:v1.12.0
gcr.io/k8s-minikube/storage-provisioner:v5
docker.io/kindest/kindnetd:v20250512-df8de77b

$ minikube image save registry.k8s.io/pause:3.10 test.tar; echo $?
0

$ file test.tar 
test.tar: empty

$ minikube image save registry.k8s.io/pause:3.10 - >test.tar

$ file test.tar 
test.tar: empty

Attach the log file

log.txt

Debugging

The issue is related to the images that come from minikube preloads. If we pull new images into minikube we can save correctly.

containerd 2.1.4

Using #21409

Testing inside the guest

Saving registry.k8s.io/pause:3.10 fails:

$ sudo ctr -n k8s.io image export - registry.k8s.io/pause:3.10 >test.tar
ctr: failed to get reader: content digest sha256:75e060e453aa927883755f715daa02fb335ea7f148a8ab249f779be796d4bb7e: not found


Pulling echo server:

```console
$ sudo ctr -n k8s.io image pull docker.io/kicbase/echo-server:1.0
docker.io/kicbase/echo server:1.0       	saved	
└──index (127ac38a2bb9)                 	already exists	
   ├──manifest (42a89d9b22e5)           	already exists	
   │  └──config (ce2d2cda2d85)          	already exists	
   └──manifest (a82eba7887a4)           	complete   	|++++++++++++++++++++++++++++++++++++++|	
      └──config (9056ab77afb8)          	complete   	|++++++++++++++++++++++++++++++++++++++|	
application/vnd.docker.distribution.manifest.list.v2+json sha256:127ac38a2bb9537b7f252addff209ea6801edcac8a92c8b1104dacd66a583ed6
Completed pull from OCI Registry (docker.io/kicbase/echo-server:1.0)	elapsed: 2.9 s	total:  4.1 Ki	(1.4 KiB/s)

Saving echo server

$ sudo ctr -n k8s.io image export echo-server-1.0.tar docker.io/kicbase/echo-server:1.0

$ tar tf echo-server-1.0.tar 
blobs/
blobs/sha256/
blobs/sha256/127ac38a2bb9537b7f252addff209ea6801edcac8a92c8b1104dacd66a583ed6
blobs/sha256/42a89d9b22e5307cb88494990d5d929c401339f508c0a7e98a4d8ac52623fc5b
blobs/sha256/ac2c07efdce850736ed8e7b9c3bc15e251d808fc43b9eeeb88797be0def23730
blobs/sha256/ce2d2cda2d858fdaea84129deb86d18e5dbf1c548f230b79fdca74cc91729d17
index.json
manifest.json
oci-layout

Saving from host

% minikube image ls | grep docker.io/kicbase/echo-server:1.0
docker.io/kicbase/echo-server:1.0

% minikube image save docker.io/kicbase/echo-server:1.0 echo-server-1.0.tar

% tar tf echo-server-1.0.tar 
blobs/
blobs/sha256/
blobs/sha256/127ac38a2bb9537b7f252addff209ea6801edcac8a92c8b1104dacd66a583ed6
blobs/sha256/42a89d9b22e5307cb88494990d5d929c401339f508c0a7e98a4d8ac52623fc5b
blobs/sha256/ac2c07efdce850736ed8e7b9c3bc15e251d808fc43b9eeeb88797be0def23730
blobs/sha256/ce2d2cda2d858fdaea84129deb86d18e5dbf1c548f230b79fdca74cc91729d17
index.json
manifest.json
oci-layout

containerd 1.17.23

Using #21405

Saving inside the guest

Saving pause image fails

$ sudo ctr -n k8s.io image export pause.tar registry.k8s.io/pause:3.10
ctr: failed to get reader: content digest sha256:75e060e453aa927883755f715daa02fb335ea7f148a8ab249f779be796d4bb7e: not found

Pulling echo server:

$ sudo ctr -n k8s.io image pull docker.io/kicbase/echo-server:1.0
docker.io/kicbase/echo-server:1.0:                                                resolved       |++++++++++++++++++++++++++++++++++++++| 
index-sha256:127ac38a2bb9537b7f252addff209ea6801edcac8a92c8b1104dacd66a583ed6:    done           |++++++++++++++++++++++++++++++++++++++| 
manifest-sha256:42a89d9b22e5307cb88494990d5d929c401339f508c0a7e98a4d8ac52623fc5b: done           |++++++++++++++++++++++++++++++++++++++| 
config-sha256:ce2d2cda2d858fdaea84129deb86d18e5dbf1c548f230b79fdca74cc91729d17:   done           |++++++++++++++++++++++++++++++++++++++| 
layer-sha256:ac2c07efdce850736ed8e7b9c3bc15e251d808fc43b9eeeb88797be0def23730:    done           |++++++++++++++++++++++++++++++++++++++| 
elapsed: 3.4 s                                                                    total:  2.4 Ki (725.0 B/s)                                       
unpacking linux/arm64/v8 sha256:127ac38a2bb9537b7f252addff209ea6801edcac8a92c8b1104dacd66a583ed6...
done: 33.646725ms	

Saving echo server:

$ sudo ctr -n k8s.io image export echo-server-1.0.tar docker.io/kicbase/echo-server:1.0

$ tar tf echo-server-1.0.tar 
blobs/
blobs/sha256/
blobs/sha256/127ac38a2bb9537b7f252addff209ea6801edcac8a92c8b1104dacd66a583ed6
blobs/sha256/42a89d9b22e5307cb88494990d5d929c401339f508c0a7e98a4d8ac52623fc5b
blobs/sha256/ac2c07efdce850736ed8e7b9c3bc15e251d808fc43b9eeeb88797be0def23730
blobs/sha256/ce2d2cda2d858fdaea84129deb86d18e5dbf1c548f230b79fdca74cc91729d17
index.json
manifest.json
oci-layout

Saving in the host

% minikube image ls | grep docker.io/kicbase/echo-server:1.0               
docker.io/kicbase/echo-server:1.0

% minikube image save docker.io/kicbase/echo-server:1.0 echo-server-1.0.tar

% tar tf echo-server-1.0.tar
blobs/
blobs/sha256/
blobs/sha256/127ac38a2bb9537b7f252addff209ea6801edcac8a92c8b1104dacd66a583ed6
blobs/sha256/42a89d9b22e5307cb88494990d5d929c401339f508c0a7e98a4d8ac52623fc5b
blobs/sha256/ac2c07efdce850736ed8e7b9c3bc15e251d808fc43b9eeeb88797be0def23730
blobs/sha256/ce2d2cda2d858fdaea84129deb86d18e5dbf1c548f230b79fdca74cc91729d17
index.json
manifest.json
oci-layout

[nirs]

/kind bug

[nirs]

/label co/runtime/containerd

[k8s-ci-robot]

@nirs: The label(s) /label co/runtime/containerd cannot be applied. These labels are supported: api-review, tide/merge-method-merge, tide/merge-method-rebase, tide/merge-method-squash, team/katacoda, refactor, ci-short, ci-extended, ci-full. Is this label configured under labels -> additional_labels or labels -> restricted_labels in plugin.yaml?

In response to this:

/label co/runtime/containerd

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[nirs]

/cc @ComradeProgrammer

[nirs]

Running with more verbose logging reveal interesting failures that are not reported to the user:

% minikube image save registry.k8s.io/pause:3.10 test.tar --alsologtostderr --v=10
I0823 22:55:30.869496   83657 out.go:360] Setting OutFile to fd 1 ...
I0823 22:55:30.870043   83657 out.go:413] isatty.IsTerminal(1) = true
I0823 22:55:30.870058   83657 out.go:374] Setting ErrFile to fd 2...
I0823 22:55:30.870063   83657 out.go:413] isatty.IsTerminal(2) = true
I0823 22:55:30.870269   83657 root.go:338] Updating PATH: /Users/nir/.minikube/bin
I0823 22:55:30.871152   83657 config.go:182] Loaded profile config "minikube": Driver=vfkit, ContainerRuntime=containerd, KubernetesVersion=v1.33.2
I0823 22:55:30.871218   83657 cache_images.go:403] Save images: ["registry.k8s.io/pause:3.10"]
I0823 22:55:30.871284   83657 config.go:182] Loaded profile config "minikube": Driver=vfkit, ContainerRuntime=containerd, KubernetesVersion=v1.33.2
I0823 22:55:30.873868   83657 ssh_runner.go:195] Run: systemctl --version
I0823 22:55:30.873891   83657 sshutil.go:53] new ssh client: &{IP:192.168.106.221 Port:22 SSHKeyPath:/Users/nir/.minikube/machines/minikube/id_rsa Username:docker}
I0823 22:55:30.907563   83657 command_runner.go:130] > systemd 256 (256.7)
I0823 22:55:30.907625   83657 command_runner.go:130] > -PAM -AUDIT -SELINUX -APPARMOR -IMA -SMACK +SECCOMP +GCRYPT -GNUTLS +OPENSSL +ACL +BLKID +CURL -ELFUTILS -FIDO2 -IDN2 -IDN +IPTC +KMOD -LIBCRYPTSETUP -LIBCRYPTSETUP_PLUGINS +LIBFDISK +PCRE2 -PWQUALITY -P11KIT -QRENCODE -TPM2 -BZIP2 +LZ4 +XZ +ZLIB -ZSTD -BPF_FRAMEWORK -XKBCOMMON -UTMP -SYSVINIT -LIBARCHIVE
I0823 22:55:30.907716   83657 containerd.go:267] Checking existence of image with name "registry.k8s.io/pause:3.10" and sha ""
I0823 22:55:30.907896   83657 ssh_runner.go:195] Run: sudo ctr -n=k8s.io images ls name==registry.k8s.io/pause:3.10
I0823 22:55:30.919657   83657 command_runner.go:130] > REF                        TYPE                                                      DIGEST                                                                  SIZE      PLATFORMS                                                                    LABELS                                                          
I0823 22:55:30.919684   83657 command_runner.go:130] > registry.k8s.io/pause:3.10 application/vnd.docker.distribution.manifest.list.v2+json sha256:ee6521f290b2168b6e0935a181d4cff9be1ac3f505666ef0e3c98fae8199917a 261.7 KiB linux/amd64,linux/arm/v7,linux/arm64,linux/ppc64le,linux/s390x,windows/amd64 io.cri-containerd.image=managed,io.cri-containerd.pinned=pinned 
I0823 22:55:30.920071   83657 cache_images.go:494] Saving image to: test.tar
I0823 22:55:30.920388   83657 vm_assets.go:164] NewFileAsset: test.tar -> /var/lib/minikube/images/test.tar
W0823 22:55:30.920399   83657 vm_assets.go:172] NewFileAsset: test.tar is an empty file!
I0823 22:55:30.920529   83657 ssh_runner.go:195] Run: sudo rm -f /var/lib/minikube/images/test.tar
I0823 22:55:30.925066   83657 containerd.go:300] Saving image registry.k8s.io/pause:3.10: /var/lib/minikube/images/test.tar
I0823 22:55:30.925128   83657 ssh_runner.go:195] Run: sudo ctr -n=k8s.io images export /var/lib/minikube/images/test.tar registry.k8s.io/pause:3.10
I0823 22:55:30.935131   83657 command_runner.go:130] ! ctr: failed to get reader: content digest sha256:75e060e453aa927883755f715daa02fb335ea7f148a8ab249f779be796d4bb7e: not found
W0823 22:55:30.935175   83657 cache_images.go:449] Failed to load cached images for profile minikube. make sure the profile is running. saving images: containerd save /var/lib/minikube/images/test.tar: ctr images export: sudo ctr -n=k8s.io images export /var/lib/minikube/images/test.tar registry.k8s.io/pause:3.10: Process exited with status 1
stdout:

stderr:
ctr: failed to get reader: content digest sha256:75e060e453aa927883755f715daa02fb335ea7f148a8ab249f779be796d4bb7e: not found
I0823 22:55:30.935189   83657 cache_images.go:457] succeeded pulling from : 
I0823 22:55:30.935192   83657 cache_images.go:458] failed pulling from : minikube

[nirs]

I see that we run pretty old version:
https://github.com/containerd/containerd/releases/tag/v1.7.23

Current version:
https://github.com/containerd/containerd/releases/tag/v1.7.28