Conversation
|
[APPROVALNOTIFIER] This PR is NOT APPROVED This pull-request has been approved by: The full list of commands accepted by this bot can be found here. DetailsNeeds approval from an approver in each of these files:Approvers can indicate their approval by writing |
|
Welcome @aditmeno! It looks like this is your first PR to openkruise/kruise 🎉 |
There was a problem hiding this comment.
Pull request overview
This PR upgrades the project to Kubernetes 1.35 (including client-go, apiserver, kubelet, controller-runtime, and Go 1.25) and adapts controllers, webhooks, utilities, and tests to the new APIs and behaviors.
Changes:
- Bumps Go toolchain to 1.25 and all
k8s.io/*dependencies to v0.35.0 / Kubernetes v1.35.0, including updating code-generator and controller-tools versions. - Fixes behavioral changes and panics introduced by the upgrade (StatefulSet revision history nil, toleration validation signatures, PVC template validation for StatefulSets, CRI image size, credential provider plugins, informer watch semantics, etc.) and refreshes generated client/informer code and CRDs.
- Improves test stability and logging/format strings (cache sync in imagejob tests, podprobe tests avoiding hanging informers, JSON expectations, hash expectations) and adds new E2E GitHub workflows for Kubernetes 1.33, 1.34, and 1.35 clusters.
Reviewed changes
Copilot reviewed 108 out of 109 changed files in this pull request and generated 3 comments.
Show a summary per file
| File | Description |
|---|---|
| test/e2e/framework/v1beta1/util.go | Updates taint helper call to new FindMatchingUntoleratedTaint signature with context and flag. |
| test/e2e/framework/v1beta1/node_util.go | Adapts taint helpers (FindMatchingUntoleratedTaint, ToleratesTaint) to new API and adds klog import. |
| test/e2e/framework/v1beta1/framework.go | Fixes Failf usage to pass message as format string for correctness with new ginkgo. |
| test/e2e/framework/v1alpha1/workloadspread_util.go | Switches e2e container images from Httpd to Nginx to align with upstream image set in k8s 1.35. |
| test/e2e/framework/v1alpha1/podunavailablebudget_util.go | Replaces Httpd images with Nginx in PDB e2e helpers to use supported images. |
| test/e2e/framework/v1alpha1/node_util.go | Same taint helper updates as v1beta1 plus klog import. |
| test/e2e/framework/v1alpha1/framework.go | Uses common.Failf with proper format string. |
| test/e2e/framework/common/util.go | Updates IsNodeUntainted to new FindMatchingUntoleratedTaint signature with context/flag. |
| test/e2e/framework/common/types.go | Repoints common image constants from removed Httpd/Redis images to Nginx/Agnhost and tweaks comments. |
| test/e2e/apps/v1beta1/cloneset.go | Adjusts image choices in CloneSet VCT update tests to use supported images (Agnhost/Nginx). |
| test/e2e/apps/v1alpha1/cloneset.go | Same image updates for v1alpha1 CloneSet tests. |
| scripts/generate_client.sh | Includes go.sum when copying to temp GOPATH, installs code-generator tools with module mode for v0.35.0, and forces module-aware codegen (GO111MODULE=on, GOFLAGS="-mod=mod"). |
| pkg/webhook/workloadspread/validating/workloadspread_validation.go | Adapts ValidateTolerations call to new signature with PodValidationOptions. |
| pkg/webhook/util/convertor/util.go | Adds VolumesFromVolumeClaimTemplates to synthesize core volumes from PVC templates for validation. |
| pkg/webhook/uniteddeployment/validating/uniteddeployment_validation.go | Updates toleration and StatefulSet template validations to new upstream signatures, including PodValidationOptions and StatefulSetValidationOptions. |
| pkg/webhook/statefulset/validating/statefulset_validation.go | Before validating pod template, injects placeholder PVC-backed volumes from VolumeClaimTemplates and uses the new validation options to match upstream StatefulSet behavior. |
| pkg/webhook/broadcastjob/mutating/broadcastjob_create_update_handler_test.go | Extends expected JSON patches to cover removal of creationTimestamp fields introduced/changed by new apimachinery defaults. |
| pkg/util/workloadspread/workloadspread_test.go | Clears the shared cache via Replace(nil, "") instead of deleting a single key to avoid GVK/key mismatches in tests. |
| pkg/util/workloadspread/workloadspread.go | Changes enableVersionedStatus to detect workload kind via type switch instead of relying on possibly-empty TypeMeta, improving behavior with objects from API/fake clients. |
| pkg/util/volumeclaimtemplate/volume_templates_hash_test.go | Updates expected VCT hash values to match new hashing behavior (likely from dependency changes). |
| pkg/util/secret/parse_test.go | Initializes keyring with BasicDockerKeyring for tests now that the production path uses external credential provider keyrings. |
| pkg/util/secret/parse.go | Switches to plugin.NewExternalCredentialProviderDockerKeyring to incorporate external credential provider plugins along with secret-based auth. |
| pkg/util/requeueduration/duration.go | Fixes Merge to treat rd2.message as an argument ("%s") rather than a format string, avoiding accidental formatting. |
| pkg/util/pods_test.go | Uses %v in failure messages to correctly format slice/struct conditions. |
| pkg/util/json_test.go | Adjusts expected JSON to reflect new defaulting/serialization behavior (metadata now empty object instead of creationTimestamp:null). |
| pkg/util/inplaceupdate/inplace_update_vertical.go | Updates comment to reflect in-place pod resize GA status in k8s 1.35; behavior remains “add/remove resources not allowed”. |
| pkg/util/inplaceupdate/inplace_update.go | Records ObservedPodGeneration into InPlaceUpdateState and adds IsUpdateProcessedByKubelet helper comparing pod ObservedGeneration with stored generation. |
| pkg/util/imagejob/imagejob_reader_test.go | Initializes logr logger to discard, starts manager earlier, waits for cache sync with timeout and Eventually rather than Sleep, making tests deterministic and less flaky. |
| pkg/util/controllerfinder/controller_finder.go | Normalizes controller references (RS/SS/Deployment/Kruise types) to use known controller kinds’ GVs instead of relying on possibly-empty object TypeMeta. |
| pkg/features/kruise_features.go | Updates documentation for InPlacePodVerticalScaling to refer to GA status in k8s 1.35 and new release blog. |
| pkg/daemon/podprobe/pod_probe_controller_test.go | Avoids real informer/watch in tests by using a simple indexer-based lister and manually updating it after fake client writes, eliminating potential hanging watches and timing issues. |
| pkg/daemon/kuberuntime/kuberuntime_container.go | Changes event recording to pass the message via format string ("%s") avoiding subtle formatting issues. |
| pkg/daemon/criruntime/imageruntime/helpers_test.go | Adapts plugin registration to the new RegisterCredentialProviderPlugins signature with extra parameters. |
| pkg/daemon/criruntime/imageruntime/cri.go | Switches from GetSize_() to GetSize() on CRI image proto to match updated API. |
| pkg/controller/workloadspread/workloadspread_controller_utils.go | Updates taint matching call to new FindMatchingUntoleratedTaint API with context and flag. |
| pkg/controller/statefulset/stateful_set_control.go | Fixes potential nil-pointer when RevisionHistoryLimit is nil by defaulting to 10 before truncating history, consistent with upstream defaults. |
| pkg/controller/statefulset/stateful_pod_control_test.go | Simplifies fatal error calls (t.Fatal(err)) for clearer failure output. |
| pkg/controller/sidecarterminator/kill_container_action.go | Uses a fixed Pod GVK when creating CRR owner references, avoiding reliance on possibly-missing TypeMeta from pods. |
| pkg/controller/resourcedistribution/resourcedistribution_controller.go | Wraps condition reason with fmt.Errorf("%s", ...) to avoid treating it as a format string. |
| pkg/controller/podprobemarker/pod_probe_marker_controller_test.go | Uses t.Fatal(err) instead of t.Fatalf(err.Error()) in tests. |
| pkg/controller/daemonset/daemonset_util.go | Updates taint predicate to new helper signature with context and boolean flag. |
| pkg/controller/daemonset/daemonset_controller.go | Adjusts predicate taint matching and node resource checks to new helper signatures and switches to kube-scheduler framework types for statuses. |
| pkg/controller/cloneset/sync/cloneset_scale_test.go | Simplifies error handling in CloneSet scaling tests with t.Fatal(err). |
| pkg/controller/cloneset/cloneset_controller_test.go | Adds explicit error check on selector validation to fail test if label selector creation fails. |
| pkg/controller/broadcastjob/broadcastjob_controller_test.go | Builds a fake client with an index on pod label key/value pairs to match new list semantics used by the controller. |
| pkg/controller/broadcastjob/broadcastjob_controller.go | Updates owner reference matching to use controllerKind.Kind, switches node fitness checks to kube-scheduler framework types, and updates taint/util calls to new signatures. |
| pkg/client/informers/externalversions/policy/v1alpha1/podunavailablebudget.go | Regenerated informer: now uses ToListWatcherWithWatchListSemantics, context-aware List/Watch functions, and context.Background() defaults. |
| pkg/client/informers/externalversions/factory.go | Regenerated factory: keeps existing behavior, updates comments and deprecation note for filtered factory. |
| pkg/client/informers/externalversions/apps/v1beta1/*.go | Regenerated informers for v1beta1 apps types (StatefulSet, SidecarSet, NodeImage, ImagePullJob, ImageListPullJob, DaemonSet, CloneSet, BroadcastJob, AdvancedCronJob) to use watch-list-aware list watchers and context-aware List/Watch. |
| pkg/client/informers/externalversions/apps/v1alpha1/*.go | Same regeneration for v1alpha1 apps types (WorkloadSpread, UnitedDeployment, StatefulSet, SidecarSet, ResourceDistribution, PodProbeMarker, PersistentPodState, NodePodProbe, NodeImage, ImagePullJob, ImageListPullJob, EphemeralJob, DaemonSet, ContainerRecreateRequest, CloneSet, BroadcastJob, AdvancedCronJob). |
| pkg/client/informers/externalversions/apps/v1alpha1/workloadspread.go | Additionally updated CRD-level toleration operator docs via CRD changes, but informer itself just uses new watcher semantics. |
| pkg/client/clientset/versioned/typed/policy/v1alpha1/policy_client.go | Generated client now uses setConfigDefaults without error return and drops error handling around it, matching new generator API. |
| pkg/client/clientset/versioned/typed/apps/v1beta1/apps_client.go | Same style change for v1beta1 apps client. |
| pkg/client/clientset/versioned/typed/apps/v1alpha1/apps_client.go | Same style change for v1alpha1 apps client. |
| pkg/client/clientset/versioned/fake/clientset_generated.go | Fake clientset updated for new tracker watch signature (takes ListOptions) and adds IsWatchListSemanticsUnSupported hint method for Reflector’s watch-list semantics. |
| go.mod | Bumps Go version to 1.25, updates Kubernetes and related dependencies to v0.35.0/v1.35.0, and aligns various indirect deps (grpc, OTel, etc.) and replace directives with the new k8s version set. |
| config/webhook/manifests.yaml | Regenerates webhook configurations, effectively reordering some webhooks and reintroducing pod mutation and specific validating hooks in updated positions. |
| config/manager/kustomization.yaml | Adds apiVersion/kind/images to the manager kustomization so the controller image name/tag can be overridden (e.g., for tests). |
| config/crd/bases/policy.kruise.io_podunavailablebudgets.yaml | Regenerated CRD with updated controller-gen version and small doc tweak (“an workload” → “a workload”). |
| config/crd/bases/apps.kruise.io_workloadspreads.yaml | Regenerated CRD: controller-gen version bump and toleration operator docs updated to include Lt and Gt. |
| config/crd/bases/apps.kruise.io_uniteddeployments.yaml | Same toleration operator doc updates and controller-gen version bump. |
| config/crd/bases/apps.kruise.io_statefulsets.yaml | Regenerated CRD with new controller-gen version. |
| config/crd/bases/apps.kruise.io_sidecarsets.yaml | Same controller-gen version update. |
| config/crd/bases/apps.kruise.io_resourcedistributions.yaml | Same controller-gen version update. |
| config/crd/bases/apps.kruise.io_podprobemarkers.yaml | Same controller-gen version update. |
| config/crd/bases/apps.kruise.io_persistentpodstates.yaml | Regenerated CRD, controller-gen version bump, and minor doc fix (“an workload” → “a workload”). |
| config/crd/bases/apps.kruise.io_nodepodprobes.yaml | Same controller-gen version update. |
| config/crd/bases/apps.kruise.io_nodeimages.yaml | Same controller-gen version update. |
| config/crd/bases/apps.kruise.io_imagepulljobs.yaml | Same controller-gen version update. |
| config/crd/bases/apps.kruise.io_imagelistpulljobs.yaml | Same controller-gen version update. |
| config/crd/bases/apps.kruise.io_ephemeraljobs.yaml | Same controller-gen version update. |
| config/crd/bases/apps.kruise.io_daemonsets.yaml | Same controller-gen version update. |
| config/crd/bases/apps.kruise.io_containerrecreaterequests.yaml | Same controller-gen version update. |
| config/crd/bases/apps.kruise.io_clonesets.yaml | Same controller-gen version update. |
| config/crd/bases/apps.kruise.io_broadcastjobs.yaml | Same controller-gen version update. |
| config/crd/bases/apps.kruise.io_advancedcronjobs.yaml | Same controller-gen version update. |
| cmd/daemon/main.go | Updates credential provider plugin registration to new function signature with additional params. |
| apis/apps/pub/inplace_update.go | Extends InPlaceUpdateState with ObservedPodGeneration to support kubelet generation tracking. |
| Makefile | Bumps envtest k8s version to 1.35.0, updates controller-gen to v0.20.0, and golangci-lint to v1.63.4. |
| Dockerfile_multiarch | Switches builder base image to golang:1.25-alpine3.21 without a fixed digest, aligning with Go 1.25 toolchain. |
| Dockerfile | Same base image bump to golang:1.25-alpine3.21 for the single-arch build. |
| .golangci.yml | Modernizes golangci-lint config: disables all linters by default, enables a small set, restructures misspell/depguard settings, and simplifies exclusions. |
| .github/workflows/e2e-1.35.yaml | New E2E workflow matrix for Kubernetes 1.35 with multiple focused jobs; currently still sets GO_VERSION to 1.23 (mismatched with go.mod). |
| .github/workflows/e2e-1.34.yaml | New E2E workflow for Kubernetes 1.34 with the same structure and the same GO_VERSION 1.23 setting. |
| .github/workflows/e2e-1.33.yaml | New E2E workflow for Kubernetes 1.33 with the same structure and GO_VERSION 1.23. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
6bf6b9d to
4dfa5ab
Compare
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 113 out of 115 changed files in this pull request and generated 1 comment.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## master #2357 +/- ##
==========================================
- Coverage 48.68% 48.43% -0.25%
==========================================
Files 324 324
Lines 27920 28072 +152
==========================================
+ Hits 13592 13597 +5
- Misses 12787 12915 +128
- Partials 1541 1560 +19
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
888b4a6 to
710786e
Compare
1dd62c8 to
77eddfa
Compare
|
@aditmeno thanks for your patch, but openkruise usually bump the kubernetes to even version number, so the next planed upgrade is for k8s 1.34 |
I see, I'll update my PR to target 1.34 instead |
146b4e8 to
a46c201
Compare
ecb2b61 to
aa79fd6
Compare
Upgrade Kubernetes dependencies from v1.32.10 to v1.34.0: - k8s.io/api, apimachinery, client-go, etc. to v0.34.0 - sigs.k8s.io/controller-runtime to v0.20.2 - Go version to 1.24.0 OpenKruise follows a convention of bumping Kubernetes to even version numbers (1.32, 1.34, 1.36, etc.). Bug fixes required by the upgrade: - Fix nil pointer dereference in StatefulSet truncateHistory when RevisionHistoryLimit is nil (default to 10) - Fix StatefulSet webhook validation to add placeholder volumes from VolumeClaimTemplates, mirroring upstream Kubernetes validation - Fix imagejob_reader_test cache synchronization using gomega.Eventually instead of time.Sleep for reliable test execution E2E test improvements: - Add robust wait functions with diagnostic output (WaitForDaemonSetUpdated, WaitForDaemonSetReady, WaitForBroadcastJobDesired, WaitForBroadcastJobSucceeded, WaitForCloneSetRunning, WaitForDeploymentRunning) - Enhance WaitForPodLabeled, WaitForPodAnnotated, WaitForState with failure diagnostics - Replace inline gomega.Eventually with robust wait functions Unit test coverage: - Add TestIsUpdateProcessedByKubelet for IsUpdateProcessedByKubelet() - Add TestVolumesFromVolumeClaimTemplates for VolumesFromVolumeClaimTemplates() - Add tests for controller finder getPod* functions Also adds E2E workflow files for Kubernetes 1.33 and 1.34. Signed-off-by: Aditya Menon <amenon@canarytechnologies.com> Add feature implementations for Kubernetes 1.34 upgrade This commit adds new features that align with Kubernetes 1.34: 1. BroadcastJob PodFailurePolicy (openkruise#2363): - Add PodFailurePolicy types to v1alpha1 and v1beta1 APIs - Add conversion functions between API versions - Add webhook validation for PodFailurePolicy rules - Add comprehensive unit tests and E2E tests 2. StatefulSet MaxUnavailable with OrderedReady (openkruise#2359): - Add MaxUnavailableStatefulSet feature gate (default enabled) - Allow maxUnavailable > 1 with OrderedReady pod management policy - Add unit tests and E2E tests for the feature 3. DaemonSet HostPort + MaxSurge Warning (openkruise#2361): - Add webhook warning when using hostPort with maxSurge > 0 - This helps users avoid potential port conflicts during rolling updates - Add comprehensive unit tests 4. Additional improvements: - Update CRD manifests for BroadcastJob PodFailurePolicy - Add E2E tests for all new features - Fix AdvancedCronJob controller to handle PodFailurePolicy Signed-off-by: Aditya Menon <amenon@canarytechnologies.com> Add BroadcastJob PodReplacementPolicy support This commit adds PodReplacementPolicy to BroadcastJob API (openkruise#2364): - Add PodReplacementPolicy type with two values: - TerminatingOrFailed: recreate pods when terminating or failed (default when no podFailurePolicy) - Failed: wait until pod is fully terminated before creating replacement (default with podFailurePolicy) - Add field to BroadcastJobSpec in both v1alpha1 and v1beta1 APIs - Add conversion functions between API versions - Add webhook validation for valid policy values - Add unit tests for validation - Add E2E tests for both policy values - Regenerate CRD manifests This matches the upstream Kubernetes Job PodReplacementPolicy (GA in k8s 1.34). Signed-off-by: Aditya Menon <amenon@canarytechnologies.com> Add tests for PodFailurePolicy TooMany validation edge cases Add test coverage for the TooMany error paths in PodFailurePolicy validation: - Too many rules (> 20) - Too many onPodConditions (> 20) - Too many exit code values (> 255) These tests improve coverage for the broadcastjob webhook validation handler. Signed-off-by: Aditya Menon <amenon@canarytechnologies.com>
@furykerry The PR is now ready for review, I've downgraded the SDK to 1.34 |
Signed-off-by: Aditya Menon <amenon@canarytechnologies.com>
Ⅰ. Describe what this PR does
This PR upgrades the Kubernetes SDK and related dependencies to version 1.34.3, following OpenKruise's convention of upgrading to even Kubernetes versions (1.32, 1.34, 1.36, etc.).
Version Changes:
API Parity Verification for K8s 1.34 GA Features:
New Features Implemented:
BroadcastJob PodFailurePolicy (Add Pod Failure Policy support to BroadcastJob #2363)
PodFailurePolicyfield to BroadcastJob API (both v1alpha1 and v1beta1)onExitCodes) and pod condition matching (onPodConditions)FailJob,Ignore,CountBroadcastJob PodReplacementPolicy (Add Pod Replacement Policy support to BroadcastJob #2364)
PodReplacementPolicyfield to BroadcastJob APITerminatingOrFailed(recreate when terminating/failed),Failed(wait until fully terminated)StatefulSet MaxUnavailable with OrderedReady (Expand Advanced StatefulSet maxUnavailable to support OrderedReady policy #2359)
MaxUnavailableStatefulSetfeature gate (enabled by default)maxUnavailable > 1withOrderedReadypod management policyDaemonSet HostPort + MaxSurge Warning (Add validation warning for DaemonSet with HostPort and maxSurge > 0 #2361)
hostPortwithmaxSurge > 0CronJob Scheduled Timestamp Annotation (Add CronJob scheduled timestamp annotation to AdvancedCronJob #2365)
batch.kubernetes.io/cronjob-scheduled-timestampannotation to AdvancedCronJobBug Fixes Required by Upgrade:
StatefulSet nil pointer fix (
pkg/controller/statefulset/stateful_set_control.go)RevisionHistoryLimitcan now be nil; added nil check with default value of 10StatefulSet webhook VolumeClaimTemplate validation (
pkg/webhook/statefulset/validating/statefulset_validation.go)Test cache sync fix (
pkg/util/imagejob/imagejob_reader_test.go)time.Sleepwithgomega.Eventuallyfor cache syncAPI compatibility fixes for K8s 1.34
FindMatchingUntoleratedTaintcalls (removed logger and bool args)TolerationsTolerateTaintcalls (removed logger and bool args)ValidateTolerationscalls (removed PodValidationOptions arg)noderesources.Fitscalls (removed nil arg)Other Changes:
Ⅱ. Does this pull request fix one issue?
Tracking issues:
Ⅲ. Describe how to verify it
Unit tests:
BroadcastJob with PodReplacementPolicy:
StatefulSet with OrderedReady and maxUnavailable:
Ⅳ. Special notes for reviews