Add topologySpreadConstraints configuration to pod spec. #2530

laiminhtrung1997 · 2024-02-04T15:34:46Z

Dear all,

I think we should configure topologySpreadConstraints to pod spec so these pods can spread zones for high availability.

Could someone review it, please? Thank you very much.

Best regards.

monotek · 2024-05-16T11:22:19Z

We need that feature too.

FxKu · 2024-06-26T13:39:50Z

pkg/cluster/cluster.go

 	}
+	if !reflect.DeepEqual(c.Statefulset.Spec.Template.Spec.TopologySpreadConstraints, statefulSet.Spec.Template.Spec.TopologySpreadConstraints) {
+		needsReplace = true
+		needsRollUpdate = true


does this really need to trigger a rolling update of pods executed by operator? Will not K8s take care of it then once the statefulset is replaced?

I see, but is this wrong too?
https://github.com/zalando/postgres-operator/blob/master/pkg/cluster/cluster.go#L472

hm good point. Maybe we can leave as is for now. With rolling update we make sure pods immediately adhere the new constraints.

FxKu · 2024-06-26T13:43:49Z

pkg/apis/acid.zalan.do/v1/postgresql_type.go

 	InitContainersOld       []v1.Container `json:"init_containers,omitempty"`
 	PodPriorityClassNameOld string         `json:"pod_priority_class_name,omitempty"`
+
+	TopologySpreadConstraints []v1.TopologySpreadConstraint `json:"topologySpreadConstraints,omitempty"`


when introducing new fields to the api you have to update the CRDs, too. Unfortunately, we don't use a modern framework where this can be generated and you have to the manifests and in go code.

Plus, you need to run code generation -> ./hack/update-codegen.sh

FxKu · 2024-06-26T13:45:26Z

pkg/cluster/k8sres.go

 		false,
 		"",
 		false,
+		[]v1.TopologySpreadConstraint{},


atm we reuse configured tolerations also for logical backup so I guess we can do the same with constraints

Ok I got it.

FxKu · 2024-06-26T13:46:17Z

pkg/cluster/k8sres.go

 	return podAntiAffinity
 }

+func generateTopologySpreadConstraints(labels labels.Set, topologySpreadConstraintObjs []v1.TopologySpreadConstraint) []v1.TopologySpreadConstraint {


would like to see a unit test for this function :)

I added a unit test for this function.

FxKu · 2024-06-26T13:47:40Z

Can you also write an e2e test that tests that the constraints work as expected, please?

laiminhtrung1997 · 2024-06-27T08:34:56Z

Can you also write an e2e test that tests that the constraints work as expected, please?

Dear @FxKu
Thanks so much for your comments, I haven't written the UT or e2e test many, but I'll try my best. I'll mark it ready and let you know when I fixed the comments.
Best regards.

FxKu · 2024-11-11T11:12:20Z

pkg/apis/acid.zalan.do/v1/crds.go

+								XPreserveUnknownFields: util.True(),
+							},
+						},
+					},


same here with the XPreserveUnknownFields. Yes there are other fields where are doing it like this, but lets get it right for new additions. I know, it's tedious to reflect the full schema because we don use a framework like kubeBulder. But it should be the trade-off for contributors when they go the "easy way" with allowing full specs in our manifest over custom stripped-down designs better suitable for end users.

FxKu · 2024-11-11T11:17:05Z

@laiminhtrung1997 have you tested how topology spread constraints behave together with specified nodeAffinity in the manifest and globally configured pod anti affinity rules. How easy is it to create a scenario where they contradict themselves and lead to scheduling problems? Should one be used over the other? Maybe @monotek can answer this, too?

FxKu · 2024-11-14T11:35:09Z

Advice for the future: Don't force push and squash your commits in the middle of a review. Now it's super hard for me to see what feedback you've reflected and I have to review everything again 😞

laiminhtrung1997 · 2024-11-15T02:12:46Z

@FxKu

It is note. I am truly sorry for this. It will not happen again.

laiminhtrung1997 · 2024-11-18T08:47:44Z

@FxKu

Currently, I have configured the operator using topologySpreadConstraints and Affinity.PodAntiAffinity together. My expectation is that the pods are always scheduled in different nodes and availability zones. This is my manifest for them.

# OperatorConfiguration
configuration:
  kubernetes:
    enable_pod_antiaffinity: true
    pod_antiaffinity_preferred_during_scheduling: false
    pod_antiaffinity_topology_key: kubernetes.io/hostname
    topology_spread_constraints:
    - max_skew: 1
      topology_key: topology.kubernetes.io/zone
      when_unsatisfiable: DoNotSchedule

Do you want me to put that scenario in the e2e test?

laiminhtrung1997 · 2025-02-25T14:30:34Z

Excuse me @FxKu

Is there any news on this?

laiminhtrung1997 · 2025-03-08T08:09:36Z

Hi @FxKu

I have fixed all your comments. Please review the PR again. I look forward to hearing from you.

Best regards.

zalando-robot · 2025-09-07T03:35:25Z

Cannot start a pipeline due to:

No accountable user for this pipeline: no Zalando employee associated to this GitHub username

Click on pipeline status check Details link below for more information.

laiminhtrung1997 requested review from sdudoladov, Jan-M, FxKu, jopadi, idanovinda, hughcapet and macedigital as code owners February 4, 2024 15:34

laiminhtrung1997 force-pushed the add-topologySpreadConstraints branch 2 times, most recently from b948c94 to 44fcb1f Compare February 4, 2024 16:16

laiminhtrung1997 mentioned this pull request Feb 4, 2024

Add topologySpreadConstraints configuration. #2474

Closed

FxKu added this to the 2.0 milestone Feb 13, 2024

FxKu modified the milestones: 2.0, 1.13.0 May 24, 2024

FxKu reviewed Jun 26, 2024

View reviewed changes

FxKu modified the milestones: 1.13.0, 1.14.0 Jun 26, 2024

laiminhtrung1997 force-pushed the add-topologySpreadConstraints branch 8 times, most recently from 530f847 to 18023cb Compare July 7, 2024 06:56

FxKu reviewed Nov 11, 2024

View reviewed changes

FxKu modified the milestones: 1.14.0, 1.15.0 Nov 11, 2024

laiminhtrung1997 requested a review from FxKu November 12, 2024 06:15

laiminhtrung1997 force-pushed the add-topologySpreadConstraints branch 3 times, most recently from 3e99f92 to 794f6db Compare November 14, 2024 10:31

laiminhtrung1997 force-pushed the add-topologySpreadConstraints branch from 27f69f5 to b592ce8 Compare February 25, 2025 14:30

laiminhtrung1997 force-pushed the add-topologySpreadConstraints branch 2 times, most recently from adfafef to 16caf11 Compare March 8, 2025 07:28

laiminhtrung1997 force-pushed the add-topologySpreadConstraints branch from 16caf11 to dbb6aaa Compare March 21, 2025 04:40

FxKu modified the milestones: 1.15.0, 1.16.0 Jul 15, 2025

laiminhtrung1997 added 7 commits September 7, 2025 10:35

Add topologySpreadConstraints configuration to pod spec.

263242a

Modify the UT.

63f8657

Correct the assert with the list of constraints.

89e2405

Fix UT

8c9840c

Run update-codegen.sh to add deepcopy for new field to the api.

ad4dd00

Reuse configured TopologySpreadConstraints for logical backup.

f52d9bc

Remove x-kubernetes-preserve-unknown-fields and XPreserveUnknownFields.

a40ba6e

laiminhtrung1997 force-pushed the add-topologySpreadConstraints branch from dbb6aaa to a40ba6e Compare September 7, 2025 03:35

FxKu modified the milestones: 1.16.0, 1.15.0 Sep 17, 2025

Add topologySpreadConstraints configuration to pod spec. #2530

Are you sure you want to change the base?

Add topologySpreadConstraints configuration to pod spec. #2530

Uh oh!

Conversation

laiminhtrung1997 commented Feb 4, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

monotek commented May 16, 2024

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

FxKu commented Jun 26, 2024

Uh oh!

laiminhtrung1997 commented Jun 27, 2024

Uh oh!

Choose a reason for hiding this comment

Uh oh!

FxKu commented Nov 11, 2024

Uh oh!

FxKu commented Nov 14, 2024

Uh oh!

laiminhtrung1997 commented Nov 15, 2024

Uh oh!

laiminhtrung1997 commented Nov 18, 2024

Uh oh!

laiminhtrung1997 commented Feb 25, 2025

Uh oh!

laiminhtrung1997 commented Mar 8, 2025

Uh oh!

zalando-robot commented Sep 7, 2025

Uh oh!

Uh oh!

laiminhtrung1997 commented Feb 4, 2024 •

edited

Loading