Use a Deployment with AntiAfinnity for kube-apiserver

Before a DaemonSet was used to deploy multiple kube-apiservers these
were bound to the hostport 6443, because DaemonSet do not support a
rollover without 1 pos bing unavailable this caused issues with one API
server endpoint becoming unavailable on update.
In systems where no network level loadbalancer of these was implemented
it causes Helm to error as it can no longer check how the rollout of
its update is going causing the kube-apiserver to never be updated.

This changes the multi controller setup to use a Deployment just like a
single controller setup. It uses Pod AntiAffinity rules to spread to all
 controller nodes.

Signed-off-by: Maartje Eyskens <maartje@eyskens.me>

assets/charts/control-plane/kube-apiserver/templates/kube-apiserver.yaml

@@ -1,4 +1,3 @@
-{{- if eq (int .Values.apiserver.replicas) 1 }}
 apiVersion: apps/v1
 kind: Deployment
 metadata:
@@ -8,7 +7,7 @@ metadata:
     tier: control-plane
     k8s-app: kube-apiserver
 spec:
-  replicas: 1
+  replicas: {{ .Values.apiserver.replicas }}
   selector:
     matchLabels:
       tier: control-plane
@@ -22,11 +21,28 @@ spec:
       labels:
         tier: control-plane
         k8s-app: kube-apiserver
+        helm-revision: {{ .Release.Revision | quote }}
       annotations:
         checkpointer.alpha.coreos.com/checkpoint: "true"
         seccomp.security.alpha.kubernetes.io/pod: 'docker/default'
         # Automatically rolls update when secret changes.
         checksum/secret: {{ include (print $.Template.BasePath "/kube-apiserver-secret.yaml") . | sha256sum }}
     spec:
+      {{- if ne (int .Values.apiserver.replicas) 1 }}
+      affinity:
+        podAntiAffinity:
+          requiredDuringSchedulingIgnoredDuringExecution:
+            - labelSelector:
+                matchExpressions:
+                  - key: k8s-app
+                    operator: In
+                    values:
+                      - kube-apiserver
+                  - key: helm-revision
+                    operator: In
+                    values:
+                      - {{ .Release.Revision | quote }}
+              topologyKey: kubernetes.io/hostname
+        {{- end }}
 {{- template "containers" . }}
-{{- end }}
+