Skip to content

test(e2e): add ZRS PV multi-zone scheduling test #1318

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
tallaxes wants to merge 2 commits into
base: tallaxes/e2e/storage
Choose a base branch
from
Draft

test(e2e): add ZRS PV multi-zone scheduling test #1318

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
7856ffe
test(integration): add ZRS PV multi-zone scheduling test
tallaxes Dec 25, 2025
6b393df
chore: make presubmit
tallaxes Dec 25, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion charts/karpenter-crd/templates/karpenter.sh_nodeclaims.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@ apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
annotations:
controller-gen.kubebuilder.io/version: v0.18.0
controller-gen.kubebuilder.io/version: v0.19.0
name: nodeclaims.karpenter.sh
spec:
group: karpenter.sh
Expand Down
2 changes: 1 addition & 1 deletion charts/karpenter-crd/templates/karpenter.sh_nodepools.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@ apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
annotations:
controller-gen.kubebuilder.io/version: v0.18.0
controller-gen.kubebuilder.io/version: v0.19.0
name: nodepools.karpenter.sh
spec:
group: karpenter.sh
Expand Down
2 changes: 2 additions & 0 deletions go.mod
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -61,6 +61,8 @@ require (
sigs.k8s.io/karpenter v1.6.2
)

replace sigs.k8s.io/karpenter => github.com/tallaxes/karpenter-core v0.0.0-20251221063144-7863910ee631
Copy link
Member

@matthchr matthchr Jan 7, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Dont forget to remove before merge.


require (
github.com/Azure/azure-sdk-for-go/sdk/internal v1.11.2 // indirect
github.com/Azure/azure-sdk-for-go/sdk/resourcemanager/compute/armcompute/v6 v6.4.0 // indirect
Expand Down
4 changes: 2 additions & 2 deletions go.sum
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -315,6 +315,8 @@ github.com/stretchr/testify v1.8.0/go.mod h1:yNjHg4UonilssWZ8iaSj1OCr/vHnekPRkoO
github.com/stretchr/testify v1.8.2/go.mod h1:w2LPCIKwWwSfY2zedu0+kehJoqGctiVI29o6fzry7u4=
github.com/stretchr/testify v1.11.1 h1:7s2iGBzp5EwR7/aIZr8ao5+dra3wiQyKjjFuvgVKu7U=
github.com/stretchr/testify v1.11.1/go.mod h1:wZwfW3scLgRK+23gO65QZefKpKQRnfz6sD981Nm4B6U=
github.com/tallaxes/karpenter-core v0.0.0-20251221063144-7863910ee631 h1:NttToNLjyuzgGanFMQfH6Sws0iqjIYeTyY98fUiSsF4=
github.com/tallaxes/karpenter-core v0.0.0-20251221063144-7863910ee631/go.mod h1:AxCaeRjv1Pgw/Ff7vT4aqyXcg8v1UdBcfzWMCaKSVjA=
github.com/tidwall/gjson v1.18.0 h1:FIDeeyB800efLX89e5a8Y0BNH+LOngJyGrIWxG2FKQY=
github.com/tidwall/gjson v1.18.0/go.mod h1:/wbyibRr2FHMks5tjHJ5F8dMZh3AcwJEMf5vlfC0lxk=
github.com/tidwall/match v1.1.1 h1:+Ho715JplO36QYgwN9PGYNhgZvoUSc9X2c80KVTi+GA=
Expand Down Expand Up @@ -494,8 +496,6 @@ sigs.k8s.io/controller-runtime v0.21.0 h1:CYfjpEuicjUecRk+KAeyYh+ouUBn4llGyDYytI
sigs.k8s.io/controller-runtime v0.21.0/go.mod h1:OSg14+F65eWqIu4DceX7k/+QRAbTTvxeQSNSOQpukWM=
sigs.k8s.io/json v0.0.0-20241010143419-9aa6b5e7a4b3 h1:/Rv+M11QRah1itp8VhT6HoVx1Ray9eB4DBr+K+/sCJ8=
sigs.k8s.io/json v0.0.0-20241010143419-9aa6b5e7a4b3/go.mod h1:18nIHnGi6636UCz6m8i4DhaJ65T6EruyzmoQqI2BVDo=
sigs.k8s.io/karpenter v1.6.2 h1:WFayZ49CSOaDMku1iYBTsD3A9hOB2yU/U95VcSAJ8KM=
sigs.k8s.io/karpenter v1.6.2/go.mod h1:AxCaeRjv1Pgw/Ff7vT4aqyXcg8v1UdBcfzWMCaKSVjA=
sigs.k8s.io/randfill v0.0.0-20250304075658-069ef1bbf016/go.mod h1:XeLlZ/jmk4i1HRopwe7/aU3H5n1zNUcX6TM94b3QxOY=
sigs.k8s.io/randfill v1.0.0 h1:JfjMILfT8A6RbawdsK2JXGBR5AQVfd+9TbzrlneTyrU=
sigs.k8s.io/randfill v1.0.0/go.mod h1:XeLlZ/jmk4i1HRopwe7/aU3H5n1zNUcX6TM94b3QxOY=
Expand Down
2 changes: 1 addition & 1 deletion pkg/apis/crds/karpenter.sh_nodeclaims.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@ apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
annotations:
controller-gen.kubebuilder.io/version: v0.18.0
controller-gen.kubebuilder.io/version: v0.19.0
name: nodeclaims.karpenter.sh
spec:
group: karpenter.sh
Expand Down
2 changes: 1 addition & 1 deletion pkg/apis/crds/karpenter.sh_nodepools.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@ apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
annotations:
controller-gen.kubebuilder.io/version: v0.18.0
controller-gen.kubebuilder.io/version: v0.19.0
name: nodepools.karpenter.sh
spec:
group: karpenter.sh
Expand Down
13 changes: 13 additions & 0 deletions test/pkg/environment/common/expectations.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -510,6 +510,19 @@ func (env *Environment) EventuallyExpectHealthyPodCountWithTimeout(timeout time.
return pods
}

// EventuallyExpectPVCBound waits for the PVC to reach Bound phase and returns its PV.
func (env *Environment) EventuallyExpectPVCBound(pvc *corev1.PersistentVolumeClaim) *corev1.PersistentVolume {
GinkgoHelper()
pv := &corev1.PersistentVolume{}
Eventually(func(g Gomega) {
g.Expect(env.Client.Get(env.Context, client.ObjectKeyFromObject(pvc), pvc)).To(Succeed())
g.Expect(pvc.Status.Phase).To(Equal(corev1.ClaimBound))
g.Expect(pvc.Spec.VolumeName).ToNot(BeEmpty())
g.Expect(env.Client.Get(env.Context, client.ObjectKey{Name: pvc.Spec.VolumeName}, pv)).To(Succeed())
}).Should(Succeed())
return pv
}

func (env *Environment) ExpectHealthyPodCount(selector labels.Selector, numPods int) []*corev1.Pod {
GinkgoHelper()
By(fmt.Sprintf("expecting %d pods matching selector %s to be ready", numPods, selector.String()))
Expand Down
41 changes: 41 additions & 0 deletions test/suites/storage/suite_test.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -224,6 +224,47 @@ var _ = Describe("Persistent Volumes", func() {
env.EventuallyExpectHealthy(pod)
env.ExpectCreatedNodeCount("==", 1)
})
// Validates the sigs.k8s.io/karpenter fix (PR kubernetes-sigs/karpenter#2743) to schedule pods
// on nodes in specific zones, even when PV is accessible in multiple zones.
It("should schedule a pod with ZRS PV when pod targets a specific zone", func() {
zones := env.GetAvailableZones()
if len(zones) == 0 {
Skip(fmt.Sprintf("skipping ZRS test because region %s does not support availability zones", env.Region))
}

// Create ZRS storage class with Immediate binding
zrsStorageClass := &storagev1.StorageClass{
ObjectMeta: metav1.ObjectMeta{
Name: "azuredisk-sc-zrs",
},
Provisioner: "disk.csi.azure.com",
Parameters: map[string]string{"skuname": "Premium_ZRS"},
VolumeBindingMode: lo.ToPtr(storagev1.VolumeBindingImmediate),
}

pvc := test.PersistentVolumeClaim(test.PersistentVolumeClaimOptions{
StorageClassName: lo.ToPtr(zrsStorageClass.Name),
})

env.ExpectCreated(nodeClass, nodePool, zrsStorageClass, pvc)
env.EventuallyExpectPVCBound(pvc)

// ZRS disks are available in all zones of the region. To test that Karpenter
// considers ALL PV node affinity terms (not just the first), we intentionally
// schedule the pod in zone 2. Without the sigs.k8s.io/karpenter fix, this would fail
// because the scheduler only looked at the first zone.
zone2 := utils.MakeAKSLabelZoneFromARMZone(env.Region, zones[1])
Copy link
Member

@matthchr matthchr Jan 7, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is it guaranteed that zones is always either len 0 or len 2? It can't be len 1?

If it's length 1, we'd get here and panic


pod := test.Pod(test.PodOptions{
PersistentVolumeClaims: []string{pvc.Name},
NodeSelector: map[string]string{corev1.LabelTopologyZone: zone2},
})

env.ExpectCreated(pod)
env.EventuallyExpectHealthy(pod)
nodes := env.ExpectCreatedNodeCount("==", 1)
Expect(nodes[0].Labels[corev1.LabelTopologyZone]).To(Equal(zone2))
})
})
})

Expand Down
Loading