Adding Tolerations feature (#10)

* Moved the packages out of pkg to simplify the structure a bit

* Fix local deployment

* WIP: adding tolerations and central config for all namespaces

* Switched to using central config for all namespaces and added tolerations.
Renamed the affinityinjector package to injector.

* Quickfix: go.mod

* Added namespace tolerations

* Tests for the injector package with tolerations

* Added a sample pod to the examples for easy testing

* - Lint fix
 - Updated readme

* Attempt to fix go.sum

* Update to go 1.17

.github/workflows/cicd.yaml

@@ -22,7 +22,7 @@ jobs:
       - name: Set up Go
         uses: actions/setup-go@v2
         with:
-          go-version: 1.16
+          go-version: 1.17
 
       - name: Install dependencies
         run: go get -u golang.org/x/lint/golint

README.md

@@ -3,7 +3,7 @@
 
 # Namespace Node Affinity
 
-Namespace Node Affinity is a Kubernetes mutating webhook which provides the ability to define node affinity for pods on a namespace level.
+Namespace Node Affinity is a Kubernetes mutating webhook which provides the ability to define node affinity and/or tolerations for pods on a namespace level.
 
 It is a replacement for the [PodNodeSelector](https://kubernetes.io/docs/reference/access-authn-authz/admission-controllers/#podnodeselector) admission controller and it is useful when using a managed k8s control plane such as [GKE](https://cloud.google.com/kubernetes-engine) or [EKS](https://aws.amazon.com/eks) where you do not have the ability to enable additional admission controller plugins and the [PodNodeSelector](https://kubernetes.io/docs/reference/access-authn-authz/admission-controllers/#podnodeselector) might not be available. The only admission controller plugin required to run the namespace-node-affinity mutating webhook is the `MutatingAdmissionWebhook` which is already enabled on most managed Kubernetes services such as [EKS](https://docs.aws.amazon.com/eks/latest/userguide/platform-versions.html).
 
@@ -18,8 +18,10 @@ kubectl apply -k deployments/base
 
 This will create the following:
  * namespace-node-affinity ServiceAccount
+ * namespace-node-affinity Role
+ * namespace-node-affinity RoleBinding
  * namespace-node-affinity ClusterRole
- * namespace-node-affinity-rolebinding ClusterRoleBinding
+ * namespace-node-affinity ClusterRoleBinding
  * namespace-node-affinity Service
  * namespace-node-affinity Deployment
 
@@ -31,11 +33,11 @@ Docker images for the webhook are available for multiple platforms [here](https:
 
 # Required Permissions
 
-The namespace-node-affinity webhook requires `get` permissions for `configmaps` in all namespaces, so it can read the configuration for each namespace it's enabled for.
+The namespace-node-affinity webhook requires `get` permissions for `configmaps` in the namespace where the centralised config is deployed.
 
 The init container (if used) requires `get`, `create` and `update` for `mutatingwebhookconfigurations` in the `admissionregistration.k8s.io` api group to create or update the MutatingWebhookConfiguration.
 
-The `ClusterRole` included in [deployments](/deployments/) already includes all of the required permissions.
+The `Role` and `ClusterRole` included in [deployments](/deployments/) already include all of the required permissions and the supplied `RoleBinding` and `ClusterRoleBinding` binds the `Role` and `ClusterRole` to the `ServiceAccount` used by the webhook.
 
 # Configuration
 
@@ -44,28 +46,37 @@ To enable the namespace-node-affinity mutating webhook on a namespace you simply
 kubectl label ns my-namespace namespace-node-affinity=enabled
 ```
 
-In order to add `nodeAffinity` to pods in that namespace you will have to create a `ConfigMap` named `namespace-node-affinity` that contains a `nodeSelectorTerms` key with the node selector terms in either JSON or YAML format. The `nodeSelectorTerms` from the config map will be added as `requiredDuringSchedulingIgnoredDuringExecution` node affinity type to each pod that is created in the labeled namespace. An example config map can be found in [examples/sample_configmap.yaml](/examples/sample_configmap.yaml). More information on how node affinity works can be found [here](https://kubernetes.io/docs/concepts/scheduling-eviction/assign-pod-node/#node-affinity).
+Each namespace with the `namespace-node-affinity=enabled` label will also need an entry in the `ConfigMap` where the configuration for the webhook is stored. The config for each namespace can be in either JSON or YAML format and must have at least one of `nodeSelectorTerms` or `tolerations`. The `nodeSelectorTerms` from the config will be added as `requiredDuringSchedulingIgnoredDuringExecution` node affinity type to each pod that is created in the labeled namespace. An example configuration can be found in [examples/sample_configmap.yaml](/examples/sample_configmap.yaml).
+
+More information on how node affinity works can be found [here](https://kubernetes.io/docs/concepts/scheduling-eviction/assign-pod-node/#node-affinity).
+More information on how taints and tolerations work can be found [here](https://kubernetes.io/docs/concepts/scheduling-eviction/taint-and-toleration/).
 
 # Failure Modes
 
 When using the provided init container to create the mutating webhook configuration, the namespace-node-affinity mutating webhook will fail silently so pods can still be created on the cluster if the webhook has been misconfigured. The affected namespace can be seen in the `AdmissionReview.Namespace`.
 
- * Missing `namespace-node-affinity` `ConfigMap` in a namespace labeled with `namespace-node-affinity=enabled`
+ * Missing `namespace-node-affinity` `ConfigMap`
 ```
 time="2021-04-10T09:35:06Z" level=info msg="Received AdmissionReview: {...}
 time="2021-04-10T09:35:06Z" level=error msg="missing configuration: configmaps \"namespace-node-affinity\" not found"
 ```
 
- * Missing `nodeSelectorTerms` from the `namespace-node-affinity` `ConfigMap`
+ * Missing entry for the namespace in the `ConfigMap`
+```
+time="2021-09-03T17:32:16Z" level=info msg="Received AdmissionReview: {...}
+time="2021-09-03T17:32:16Z" level=error msg="missing configuration: for testing-ns-e"
+```
+
+ * Both `nodeSelectorTerms` and `tolerations` are missing from the entry for the namespace in the `ConfigMap`
 ```
-time="2021-04-10T09:37:57Z" level=info msg="Received AdmissionReview: {...}
-time="2021-04-10T09:37:57Z" level=error msg="missing nodeSelectorTerms from config: nodeSelectorTerms is missing from the config map"
+time="2021-09-03T17:38:46Z" level=info msg="Received AdmissionReview: {...}
+time="2021-09-03T17:38:46Z" level=error msg="invalid configuration: at least one of nodeSelectorTerms or tolerations needs to be specified for testing-ns-d"
 ```
 
- * Invalid `nodeSelectorTerms` in the `namespace-node-affinity` `ConfigMap`
+ * Invalid `nodeSelectorTerms` or `tolerations` in the `namespace-node-affinity` `ConfigMap`
 ```
 time="2021-04-10T09:40:59Z" level=info msg="Received AdmissionReview: {...}
-time="2021-04-10T09:40:59Z" level=error msg="invalid configuration: error unmarshaling JSON: while decoding JSON: json: cannot unmarshal string into Go value of type []v1.NodeSelectorTerm"
+time="2021-04-10T09:40:59Z" level=error msg="invalid configuration: error unmarshaling JSON: while decoding JSON: json: cannot unmarshal string into Go struct field NamespaceConfig.nodeSelectorTerms of type []v1.NodeSelectorTerm"
 ```
 
 # Contributing

cmd/createcerts/main.go

@@ -8,13 +8,14 @@ import (
 	"crypto/x509/pkix"
 	"encoding/pem"
 	"fmt"
-	log "github.com/sirupsen/logrus"
 	"math/big"
 	"os"
 	"path/filepath"
 	"time"
 
-	"github.com/idgenchev/namespace-node-affinity/pkg/webhookconfig"
+	log "github.com/sirupsen/logrus"
+
+	"github.com/idgenchev/namespace-node-affinity/webhookconfig"
 	"github.com/jessevdk/go-flags"
 	k8sclient "k8s.io/client-go/kubernetes"
 	"k8s.io/client-go/rest"

cmd/nsnodeaffinity/main.go

@@ -6,7 +6,8 @@ import (
 	"net/http"
 	"time"
 
-	"github.com/idgenchev/namespace-node-affinity/pkg/affinityinjector"
+	"github.com/idgenchev/namespace-node-affinity/injector"
+
 	"github.com/jessevdk/go-flags"
 	"github.com/prometheus/client_golang/prometheus/promhttp"
 	log "github.com/sirupsen/logrus"
@@ -20,7 +21,8 @@ var opts struct {
 	WriteTimeout  time.Duration `long:"write-timeout" default:"10s" description:"Write timeout"`
 	CertFile      string        `lond:"cert" short:"c" env:"CERT" default:"/etc/webhook/certs/tls.crt" description:"Path to the cert file"`
 	KeyFile       string        `lond:"key" short:"k" env:"KEY" default:"/etc/webhook/certs/tls.key" description:"Path to the key file"`
-	ConfigMapName string        `long:"config-map-name" short:"m" env:"CONFIG_MAP_NAME" default:"namespace-node-affinity" description:"Name of the configm map containing the node selector terms to be applied to every pod on creation. This config map should be present in every namespace where this webhook is enabled"`
+	Namespace     string        `long:"namespace" short:"n" env:"NAMESPACE" description:"The namespace where the configmap is deployed"`
+	ConfigMapName string        `long:"config-map-name" short:"m" env:"CONFIG_MAP_NAME" default:"namespace-node-affinity" description:"Name of the configm map containing the node selector terms to be applied to every pod on creation."`
 }
 
 type injectorInterface interface {
@@ -68,7 +70,7 @@ func main() {
 	}
 
 	h := handler{
-		affinityinjector.NewAffinityInjector(clientset, opts.ConfigMapName),
+		injector.NewInjector(clientset, opts.Namespace, opts.ConfigMapName),
 	}
 	mux.HandleFunc("/mutate", h.mutate)
 

deployments/base/clusterrole.yaml

@@ -0,0 +1,9 @@
+---
+kind: ClusterRole
+apiVersion: rbac.authorization.k8s.io/v1
+metadata:
+  name: namespace-node-affinity
+rules:
+- apiGroups: ["admissionregistration.k8s.io"]
+  resources: ["mutatingwebhookconfigurations"]
+  verbs: ["get", "create", "update"]

deployments/base/clusterrolebinding.yaml

@@ -0,0 +1,14 @@
+---
+kind: ClusterRoleBinding
+apiVersion: rbac.authorization.k8s.io/v1
+metadata:
+  name: namespace-node-affinity
+subjects:
+- kind: ServiceAccount
+  name: namespace-node-affinity
+  namespace: default
+  apiGroup: ""
+roleRef:
+  kind: ClusterRole
+  name: namespace-node-affinity
+  apiGroup: rbac.authorization.k8s.io

deployments/base/deployment.yaml

@@ -1,3 +1,4 @@
+---
 apiVersion: apps/v1
 kind: Deployment
 metadata:
@@ -26,9 +27,13 @@ spec:
             memory: 64Mi
         env:
           - name: CERT
-            value: &cert /etc/webhook/certs/tls.crt
+            value: /etc/webhook/certs/tls.crt
           - name: KEY
-            value: &key /etc/webhook/certs/tls.key
+            value: /etc/webhook/certs/tls.key
+          - name: NAMESPACE
+            valueFrom:
+              fieldRef:
+                fieldPath: metadata.namespace
       initContainers:
       - name: init-webhook
         image: idgenchev/namespace-node-affinity-init-container
@@ -43,9 +48,9 @@ spec:
           - name: SERVICE_NAME
             value: namespace-node-affinity
           - name: CERT
-            value: *cert
+            value: /etc/webhook/certs/tls.crt
           - name: KEY
-            value: *key
+            value: /etc/webhook/certs/tls.key
       volumes:
       - name: webhook-certs
         emptyDir: {}

deployments/base/kustomization.yaml

@@ -1,3 +1,4 @@
+---
 apiVersion: kustomize.config.k8s.io/v1beta1
 kind: Kustomization
 
@@ -8,5 +9,7 @@ resources:
 - deployment.yaml
 - role.yaml
 - rolebinding.yaml
+- clusterrole.yaml
+- clusterrolebinding.yaml
 - sa.yaml
 - service.yaml

deployments/base/role.yaml

@@ -1,11 +1,10 @@
-kind: ClusterRole
+---
+kind: Role
 apiVersion: rbac.authorization.k8s.io/v1
 metadata:
   name: namespace-node-affinity
+  namespace: default
 rules:
-- apiGroups: ["admissionregistration.k8s.io"]
-  resources: ["mutatingwebhookconfigurations"]
-  verbs: ["get", "create", "update"]
 - apiGroups: [""]
   resources: ["configmaps"]
   verbs: ["get"]

deployments/base/rolebinding.yaml

@@ -1,13 +1,15 @@
-kind: ClusterRoleBinding
+---
+kind: RoleBinding
 apiVersion: rbac.authorization.k8s.io/v1
 metadata:
   name: namespace-node-affinity
+  namespace: default
 subjects:
 - kind: ServiceAccount
   name: namespace-node-affinity
   namespace: default
   apiGroup: ""
 roleRef:
-  kind: ClusterRole
+  kind: Role
   name: namespace-node-affinity
   apiGroup: rbac.authorization.k8s.io

deployments/base/sa.yaml

@@ -1,3 +1,4 @@
+---
 apiVersion: v1
 kind: ServiceAccount
 metadata:

deployments/base/service.yaml

@@ -1,3 +1,4 @@
+---
 apiVersion: v1
 kind: Service
 metadata:

deployments/overlays/local/deployment.yaml

@@ -1,3 +1,4 @@
+---
 apiVersion: apps/v1
 kind: Deployment
 metadata:

deployments/overlays/local/kustomization.yaml

@@ -1,3 +1,4 @@
+---
 apiVersion: kustomize.config.k8s.io/v1beta1
 kind: Kustomization
 

examples/example_namespaces.yaml

@@ -0,0 +1,35 @@
+---
+apiVersion: v1
+kind: Namespace
+metadata:
+  name: testing-ns
+  labels:
+    namespace-node-affinity: enabled
+---
+apiVersion: v1
+kind: Namespace
+metadata:
+  name: testing-ns-b
+  labels:
+    namespace-node-affinity: enabled
+---
+apiVersion: v1
+kind: Namespace
+metadata:
+  name: testing-ns-c
+  labels:
+    namespace-node-affinity: enabled
+---
+apiVersion: v1
+kind: Namespace
+metadata:
+  name: testing-ns-d
+  labels:
+    namespace-node-affinity: enabled
+---
+apiVersion: v1
+kind: Namespace
+metadata:
+  name: testing-ns-e
+  labels:
+    namespace-node-affinity: enabled

examples/sample_configmap.yaml

@@ -1,12 +1,35 @@
+---
 apiVersion: v1
 kind: ConfigMap
 metadata:
   name: namespace-node-affinity
-  namespace: testing-ns
+  namespace: default
 data:
-  nodeSelectorTerms: |
-    - matchExpressions:
-      - key: the-testing-key
-        operator: In
-        values:
-        - the-testing-val1
+  testing-ns: |
+    nodeSelectorTerms:
+      - matchExpressions:
+        - key: the-testing-key
+          operator: In
+          values:
+          - the-testing-val1
+    tolerations:
+      - key: "example-key"
+        operator: "Exists"
+        effect: "NoSchedule"
+  testing-ns-b: |
+    nodeSelectorTerms:
+      - matchExpressions:
+        - key: the-testing-key
+          operator: In
+          values:
+          - the-testing-val1
+  testing-ns-c: |
+    tolerations:
+      - key: "example-key"
+        operator: "Exists"
+        effect: "NoSchedule"
+  testing-ns-d: |
+    invalid:
+      - key: "example-key"
+        operator: "Exists"
+        effect: "NoSchedule"

examples/sample_pod.yaml

@@ -0,0 +1,16 @@
+apiVersion: v1
+kind: Pod
+metadata:
+  creationTimestamp: null
+  labels:
+    run: nginx-test
+  name: nginx-test
+  namespace: testing-ns
+spec:
+  containers:
+  - image: nginx
+    name: nginx-test
+    resources: {}
+  dnsPolicy: ClusterFirst
+  restartPolicy: Always
+status: {}