feat(dp): handle PodDisruptionBudget reconciliation for deployment

[czeslavo]

What this PR does / why we need it:

Adds PodDisruptionBudget reconciliation for DataPlanes.

Which issue this PR fixes

Fixes #142.

PR Readiness Checklist:

Complete these before marking the PR as ready to review:

• the CHANGELOG.md release notes have been updated to reflect significant changes

[pmalek]

The E2E are failing due to policy rules not being in place in the helm chart (a sad reality of chicken and egg problem in tests :/)

2024-08-08T14:05:16.22262256Z stderr F W0808 14:05:16.222529       1 reflector.go:547] pkg/mod/k8s.io/client-go@v0.30.3/tools/cache/reflector.go:232: failed to list *v1.PodDisruptionBudget: poddisruptionbudgets.policy is forbidden: User "system:serviceaccount:5670ced5-4151-4c84-8449-a08266eb6387:controller-manager" cannot list resource "poddisruptionbudgets" in API group "policy" at the cluster scope
2024-08-08T14:05:16.222648388Z stderr F E0808 14:05:16.222563       1 reflector.go:150] pkg/mod/k8s.io/client-go@v0.30.3/tools/cache/reflector.go:232: Failed to watch *v1.PodDisruptionBudget: failed to list *v1.PodDisruptionBudget: poddisruptionbudgets.policy is forbidden: User "system:serviceaccount:5670ced5-4151-4c84-8449-a08266eb6387:controller-manager" cannot list resource "poddisruptionbudgets" in API group "policy" at the cluster scope
2024-08-08T14:05:16.319688827Z stderr F {"level":"info","ts":"2024-08-08T14:05:16Z","msg":"Starting workers","controller":"gatewayclass","controllerGroup":"gateway.networking.k8s.io","controllerKind":"GatewayClass","worker count":1}
2024-08-08T14:05:17.816154465Z stderr F W0808 14:05:17.816064       1 reflector.go:547] pkg/mod/k8s.io/client-go@v0.30.3/tools/cache/reflector.go:232: failed to list *v1.PodDisruptionBudget: poddisruptionbudgets.policy is forbidden: User "system:serviceaccount:5670ced5-4151-4c84-8449-a08266eb6387:controller-manager" cannot list resource "poddisruptionbudgets" in API group "policy" at the cluster scope
2024-08-08T14:05:17.816173621Z stderr F E0808 14:05:17.816095       1 reflector.go:150] pkg/mod/k8s.io/client-go@v0.30.3/tools/cache/reflector.go:232: Failed to watch *v1.PodDisruptionBudget: failed to list *v1.PodDisruptionBudget: poddisruptionbudgets.policy is forbidden: User "system:serviceaccount:5670ced5-4151-4c84-8449-a08266eb6387:controller-manager" cannot list resource "poddisruptionbudgets" in API group "policy" at the cluster scope
2024-08-08T14:05:19.759403497Z stderr F W0808 14:05:19.759318       1 reflector.go:547] pkg/mod/k8s.io/client-go@v0.30.3/tools/cache/reflector.go:232: failed to list *v1.PodDisruptionBudget: poddisruptionbudgets.policy is forbidden: User "system:serviceaccount:5670ced5-4151-4c84-8449-a08266eb6387:controller-manager" cannot list resource "poddisruptionbudgets" in API group "policy" at the cluster scope
2024-08-08T14:05:19.759425408Z stderr F E0808 14:05:19.759349       1 reflector.go:150] pkg/mod/k8s.io/client-go@v0.30.3/tools/cache/reflector.go:232: Failed to watch *v1.PodDisruptionBudget: failed to list *v1.PodDisruptionBudget: poddisruptionbudgets.policy is forbidden: User "system:serviceaccount:5670ced5-4151-4c84-8449-a08266eb6387:controller-manager" cannot list resource "poddisruptionbudgets" in API group "policy" at the cluster scope
2024-08-08T14:05:25.542095916Z stderr F W0808 14:05:25.542015       1 reflector.go:547] pkg/mod/k8s.io/client-go@v0.30.3/tools/cache/reflector.go:232: failed to list *v1.PodDisruptionBudget: poddisruptionbudgets.policy is forbidden: User "system:serviceaccount:5670ced5-4151-4c84-8449-a08266eb6387:controller-manager" cannot list resource "poddisruptionbudgets" in API group "policy" at the cluster scope
2024-08-08T14:05:25.542113007Z stderr F E0808 14:05:25.542046       1 reflector.go:150] pkg/mod/k8s.io/client-go@v0.30.3/tools/cache/reflector.go:232: Failed to watch *v1.PodDisruptionBudget: failed to list *v1.PodDisruptionBudget: poddisruptionbudgets.policy is forbidden: User "system:serviceaccount:5670ced5-4151-4c84-8449-a08266eb6387:controller-manager" cannot list resource "poddisruptionbudgets" in API group "policy" at the cluster scope
2024-08-08T14:05:37.972238936Z stderr F W0808 14:05:37.972165       1 reflector.go:547] pkg/mod/k8s.io/client-go@v0.30.3/tools/cache/reflector.go:232: failed to list *v1.PodDisruptionBudget: poddisruptionbudgets.policy is forbidden: User "system:serviceaccount:5670ced5-4151-4c84-8449-a08266eb6387:controller-manager" cannot list resource "poddisruptionbudgets" in API group "policy" at the cluster scope
2024-08-08T14:05:37.972260346Z stderr F E0808 14:05:37.972194       1 reflector.go:150] pkg/mod/k8s.io/client-go@v0.30.3/tools/cache/reflector.go:232: Failed to watch *v1.PodDisruptionBudget: failed to list *v1.PodDisruptionBudget: poddisruptionbudgets.policy is forbidden: User "system:serviceaccount:5670ced5-4151-4c84-8449-a08266eb6387:controller-manager" cannot list resource "poddisruptionbudgets" in API group "policy" at the cluster scope
2024-08-08T14:05:51.355521461Z stderr F W0808 14:05:51.355402       1 reflector.go:547] pkg/mod/k8s.io/client-go@v0.30.3/tools/cache/reflector.go:232: failed to list *v1.PodDisruptionBudget: poddisruptionbudgets.policy is forbidden: User "system:serviceaccount:5670ced5-4151-4c84-8449-a08266eb6387:controller-manager" cannot list resource "poddisruptionbudgets" in API group "policy" at the cluster scope
2024-08-08T14:05:51.355544163Z stderr F E0808 14:05:51.355438       1 reflector.go:150] pkg/mod/k8s.io/client-go@v0.30.3/tools/cache/reflector.go:232: Failed to watch *v1.PodDisruptionBudget: failed to list *v1.PodDisruptionBudget: poddisruptionbudgets.policy is forbidden: User "system:serviceaccount:5670ced5-4151-4c84-8449-a08266eb6387:controller-manager" cannot list resource "poddisruptionbudgets" in API group "policy" at the cluster scope
2024-08-08T14:06:40.474584374Z stderr F W0808 14:06:40.474497       1 reflector.go:547] pkg/mod/k8s.io/client-go@v0.30.3/tools/cache/reflector.go:232: failed to list *v1.PodDisruptionBudget: poddisruptionbudgets.policy is forbidden: User "system:serviceaccount:5670ced5-4151-4c84-8449-a08266eb6387:controller-manager" cannot list resource "poddisruptionbudgets" in API group "policy" at the cluster scope
2024-08-08T14:06:40.474609751Z stderr F E0808 14:06:40.474523       1 reflector.go:150] pkg/mod/k8s.io/client-go@v0.30.3/tools/cache/reflector.go:232: Failed to watch *v1.PodDisruptionBudget: failed to list *v1.PodDisruptionBudget: poddisruptionbudgets.policy is forbidden: User "system:serviceaccount:5670ced5-4151-4c84-8449-a08266eb6387:controller-manager" cannot list resource "poddisruptionbudgets" in API group "policy" at the cluster scope
2024-08-08T14:07:16.212545541Z stderr F {"level":"error","ts":"2024-08-08T14:07:16Z","msg":"Could not wait for Cache to sync","controller":"dataplane","controllerGroup":"gateway-operator.konghq.com","controllerKind":"DataPlane","error":"failed to wait for dataplane caches to sync: timed out waiting for cache to be synced for Kind *v1beta1.DataPlane","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.1\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.18.4/pkg/internal/controller/controller.go:198\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.18.4/pkg/internal/controller/controller.go:203\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.18.4/pkg/internal/controller/controller.go:229\nsigs.k8s.io/controller-runtime/pkg/manager.(*runnableGroup).reconcile.func1\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.18.4/pkg/manager/runnable_group.go:226"}

I'm not sure if we want to solve it differently than adding rules from this PR to helm chart already.

We could be tempted to patch the policy rules in tests but that can be error prone.

I'm curious to your thoughts on this.

[czeslavo]

We could be tempted to patch the policy rules in tests but that can be error prone.

Yeah, I agree that's better to add the rules already to the helm chart. At least by having these E2Es failing, it gives us direct feedback on the missing pieces and we're fixing them in the right place. By patching them, we'd still have to remember to add the rules to the chart eventually.

Adding them and releasing straight away: Kong/charts#1114