iso: bump to containerd/nerdctl 2.x #21409

nirs · 2025-08-24T12:06:05Z

This is #21368 with the following changes:

Rebase on master
Fix whitespace in hash files
Unify containerd-bin makefiles
Update contained-bin Config.in
Remove containerd.conf.d directory
Add missing _AARCH64 to variable names
Bump nerdctl to 2.1.3 (it must be upgraded with containerd)

Must be merged together with #21643

Fixes #20497

k8s-ci-robot · 2025-08-24T12:06:07Z

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

k8s-ci-robot · 2025-08-24T12:06:11Z

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: nirs
Once this PR has been reviewed and has the lgtm label, please assign spowelljr for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

nirs · 2025-08-24T12:06:26Z

/ok-to-build-iso

nirs · 2025-08-24T13:42:28Z

/ok-to-build-iso

nirs · 2025-08-24T17:10:13Z

/ok-to-build-iso

deploy/iso/minikube-iso/arch/aarch64/package/containerd-bin-aarch64/containerd-bin.mk

nirs · 2025-08-24T17:26:42Z

/ok-to-build-iso

medyagh · 2025-08-25T18:28:18Z

See the logs at:
https://storage.googleapis.com/minikube-builds/logs/21409/5ca83e9/iso_build.txt

for example for this PR
https://storage.googleapis.com/minikube-builds/logs/20669/22ed44e/iso_build.txt

5ca83e9

medyagh · 2025-08-25T18:37:54Z

/ok-to-build-iso

minikube-bot · 2025-08-25T23:05:41Z

Hi @nirs, we have updated your PR with the reference to newly built ISO. Pull the changes locally if you want to test with them or update your PR further.

nirs

The config.toml does not help with #21408 and may break code configuring containerd using sed(!?). We need to replace the configuration code with proper toml parsing but for now we can keep the existing config.

deploy/iso/minikube-iso/arch/aarch64/package/containerd-bin-aarch64/containerd-bin.mk

medyagh · 2025-08-28T18:39:13Z

/ok-to-test

nirs · 2025-08-31T21:52:34Z

/ok-to-build-iso

nirs · 2025-09-30T17:46:30Z

https://storage.googleapis.com/minikube-builds/logs/21409/e4af2e4/iso_build.txt

nirs · 2025-09-30T18:48:07Z

/ok-to-build-iso

minikube-bot · 2025-09-30T22:38:27Z

Hi @nirs, we have updated your PR with the reference to newly built ISO. Pull the changes locally if you want to test with them or update your PR further.

- Fix Makefile title in aarch64 - Unify whitespaec in *.mk files - Remove extra space between the hash and the tarball name - Remove empty line at the end hash file

Based on crictl-bin Config.in.

Containerd supports a simple and poorly documented drop-in configuration files using the imports option. If this option is set: imports = ["/etc/containerd/conf.d/*.toml"] Then toml files under /etc/containerd/conf.d/ are loaded and merged with the config read from /etc/containerd/config.toml. Unlike systemd drop-in files, you cannot override single option by adding drop-in configuration file. To override a single option you must set all options in the section, and the entire section is replaced by the drop-in file. We never used this feature since our config contains: # imports And we don't configure this dynamically. Our config on the host is: $ grep conf.d /etc/containerd/config.toml conf_dir = "/etc/cni/net.d" However we were creating: /etc/containerd/containerd.conf.d/ This path does not make sense (repeating containerd twice) and files in this directory are ignored. Finally this directory was created in CONFIGURE_CMDS instead of INSTALL_CMDS. Now that we install a binary we should not have any configure commands. Since we never had a working conf.d directory we can safely remove it.

This is the reason for the strange failure when build the x86_64 iso, about no hash for arm64 tarball. I seems that package for different architectures must have a different names to avoid confusing buildroot. The name was broken by mistake when updating to containerd 2.1.4. With this change iso build works for both aarch64 and x86_64.

For containerd 2.1.4, the recommended nerdctl version is 2.1.3 or higher. Here's why this is the recommended version and where you can find it: - Version alignment: Major releases of containerd are often paired with a corresponding nerdctl release to ensure compatibility with new features and breaking changes. nerdctl 2.1.x is specifically developed and tested for the containerd 2.1.x release cycle. - Release verification: The official nerdctl release page on GitHub explicitly states the intended compatibility. For instance, the release notes for nerdctl 2.1.3 confirm it is expected to be used with containerd 2.1. - Feature support: nerdctl 2.1.x includes updates that support the latest features and architectural changes in containerd 2.1, such as UserNS-Remap mode. (From Google AI Mode) Generated using: make update-nerdctl-version and removing the kicbase change, since nerdctl 2.1.3 does not work with containerd 1.7 consumed from Ubuntu. It will be updated when we switch to newer version of Ubuntu or Debian.

minikube-pr-bot · 2025-10-18T14:45:53Z

kvm2 driver with docker runtime

┌────────────────┬──────────┬────────────────────────┐
│    COMMAND     │ MINIKUBE │ MINIKUBE  ( PR 21409 ) │
├────────────────┼──────────┼────────────────────────┤
│ minikube start │ 47.2s    │ 46.4s                  │
│ enable ingress │ 20.7s    │ 19.8s                  │
└────────────────┴──────────┴────────────────────────┘

Times for minikube start: 45.4s 46.2s 47.8s 48.9s 47.5s
Times for minikube (PR 21409) start: 45.6s 45.1s 46.0s 48.2s 47.2s

Times for minikube ingress: 21.5s 18.4s 21.5s 21.0s 20.9s
Times for minikube (PR 21409) ingress: 17.9s 17.4s 21.5s 20.9s 21.4s

docker driver with docker runtime

┌────────────────┬──────────┬────────────────────────┐
│    COMMAND     │ MINIKUBE │ MINIKUBE  ( PR 21409 ) │
├────────────────┼──────────┼────────────────────────┤
│ minikube start │ 24.8s    │ 23.5s                  │
│ enable ingress │ 12.3s    │ 12.7s                  │
└────────────────┴──────────┴────────────────────────┘

Times for minikube start: 27.2s 24.6s 24.6s 24.2s 23.5s
Times for minikube (PR 21409) start: 22.7s 23.1s 26.1s 22.3s 23.5s

Times for minikube ingress: 13.7s 10.7s 12.7s 12.7s 11.8s
Times for minikube (PR 21409) ingress: 13.7s 10.7s 12.7s 13.7s 12.7s

docker driver with containerd runtime

┌────────────────┬──────────┬────────────────────────┐
│    COMMAND     │ MINIKUBE │ MINIKUBE  ( PR 21409 ) │
├────────────────┼──────────┼────────────────────────┤
│ minikube start │ 20.9s    │ 21.9s                  │
│ enable ingress │ 21.8s    │ 21.0s                  │
└────────────────┴──────────┴────────────────────────┘

Times for minikube (PR 21409) start: 21.8s 23.6s 21.2s 20.6s 22.4s
Times for minikube start: 20.3s 20.4s 20.9s 21.9s 21.0s

Times for minikube ingress: 22.2s 22.2s 20.2s 22.2s 22.2s
Times for minikube (PR 21409) ingress: 22.2s 20.2s 20.2s 20.2s 22.2s

minikube-pr-bot · 2025-10-18T18:34:44Z

Here are the number of top 10 failed tests in each environments with lowest flake rate.

Environment	Test Name	Flake Rate
none_Linux (9 failed)	TestAddons/serial/Volcano(gopogh)	0.00% (chart)
none_Linux (9 failed)	TestAddons/parallel/CSI(gopogh)	0.00% (chart)
none_Linux (9 failed)	TestAddons/parallel/Yakd(gopogh)	0.00% (chart)
none_Linux (9 failed)	TestFunctional/parallel/ServiceCmd/DeployApp(gopogh)	0.00% (chart)
none_Linux (9 failed)	TestFunctional/parallel/ServiceCmd/HTTPS(gopogh)	0.00% (chart)
none_Linux (9 failed)	TestFunctional/parallel/ServiceCmd/Format(gopogh)	0.00% (chart)
none_Linux (9 failed)	TestFunctional/parallel/ServiceCmd/URL(gopogh)	0.00% (chart)
none_Linux (9 failed)	TestFunctional/parallel/PersistentVolumeClaim(gopogh)	0.00% (chart)
none_Linux (9 failed)	TestFunctional/parallel/MySQL(gopogh)	0.00% (chart)

Besides the following environments also have failed tests:

Docker_Linux_containerd: 17 failed (gopogh)
Docker_Linux_crio_arm64: 45 failed (gopogh)
KVM_Linux_crio: 14 failed (gopogh)
Docker_Linux_crio: 41 failed (gopogh)

To see the flake rates of all tests by environment, click here.

k8s-ci-robot · 2025-10-23T18:55:54Z

@nirs: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
pull-minikube-integration	`432f5d8`	link	true	`/test pull-minikube-integration`
integration-kvm-crio-linux-x86-64	`432f5d8`	link	true	`/test integration-kvm-crio-linux-x86-64`
integration-docker-docker-linux-x86-64	`432f5d8`	link	true	`/test integration-docker-docker-linux-x86-64`
integration-docker-containerd-linux-x86-64	`432f5d8`	link	true	`/test integration-docker-containerd-linux-x86-64`
integration-kvm-containerd-linux-x86-64	`432f5d8`	link	true	`/test integration-kvm-containerd-linux-x86-64`
integration-none-docker-linux-x86-64	`432f5d8`	link	true	`/test integration-none-docker-linux-x86-64`
integration-kvm-docker-linux-x86-64	`432f5d8`	link	true	`/test integration-kvm-docker-linux-x86-64`
integration-docker-crio-linux-x86-64	`432f5d8`	link	true	`/test integration-docker-crio-linux-x86-64`

Full PR test history. Your PR dashboard. Please help us cut down on flakes by linking to an open issue when you hit one in your PR.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

k8s-ci-robot added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Aug 24, 2025

k8s-ci-robot requested a review from ComradeProgrammer August 24, 2025 12:06

k8s-ci-robot requested a review from prezha August 24, 2025 12:06

k8s-ci-robot added cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. size/L Denotes a PR that changes 100-499 lines, ignoring generated files. labels Aug 24, 2025

nirs commented Aug 24, 2025

View reviewed changes

deploy/iso/minikube-iso/arch/aarch64/package/containerd-bin-aarch64/containerd-bin.mk Outdated Show resolved Hide resolved

nirs force-pushed the containerd2 branch from b1b7860 to 5ca83e9 Compare August 24, 2025 17:25

nirs marked this pull request as ready for review August 24, 2025 19:06

k8s-ci-robot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Aug 24, 2025

k8s-ci-robot requested a review from medyagh August 24, 2025 19:06

nirs mentioned this pull request Aug 24, 2025

containerd: preloaded images fail to save #21408

Open

nirs marked this pull request as draft August 26, 2025 21:56

k8s-ci-robot added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Aug 26, 2025

nirs commented Aug 26, 2025

View reviewed changes

deploy/iso/minikube-iso/arch/aarch64/package/containerd-bin-aarch64/containerd-bin.mk Show resolved Hide resolved

k8s-ci-robot added the ok-to-test Indicates a non-member PR verified by an org member that is safe to test. label Aug 28, 2025