Policies fails on Istio injected containers (istio-init and istio-proxy) - how to exclude containers with specific names

[JesperBerggren]

Summary: Is it possible to make a exclude a policy from a container with a specific name?

Hey everbody!

I hope I can get some help.
I've applied the restricted pod policies on a running cluster with a lot of pods running. Now I'm trying to fix all the validation errors.

Our cluster is running with Istio and of course auto-injection of sidecars, which happens with a mutating webhook admission controller from Istio.

As an example I have the policy disallow-capabilities-strict and in my Deployment of a given API I have the following configuration (in a slightly masked edition marked by ...):

apiVersion: apps/v1
kind: Deployment
metadata:
  name: digitalpost-xyz-modtagmemo-service
  namespace: digitalpost-xyz
spec:
  replicas: 1
  selector:
    matchLabels:
      app: digitalpost-xyz-modtagmemo-service
      version: v1
  template:
    metadata:
      labels:
        app: digitalpost-xyz-modtagmemo-service
        version: v1
      annotations:
        prometheus.io/scrape: "true"
        ...
    spec:
      serviceAccountName: digitalpost-xyz-modtagmemo-service
      ...
      securityContext:
        runAsNonRoot: true
        seccompProfile:
          type: RuntimeDefault
      containers:
      - name: digitalpost-modtagmemo
        ...
        ports:
        - containerPort: 8080
        resources:
          limits:
            cpu: 800m
          requests:
            cpu: 50m
        volumeMounts:
        - name: dpdir
          ...
        securityContext:
          capabilities:
            drop:
              - ALL
            add: ['NET_BIND_SERVICE']
          allowPrivilegeEscalation: false
      volumes:
       - name: dpdir
         ...

As you can see I have two securityContext, one for the pod and one for the container.
This makes it "work" and I can deploy the Deployment without any validation errors, but the replicaset is not allowed to spin up a pod by Kyverno because of the Istio containers which are injected via the mutating webhook admission controller.

So my question is: Is it possible to make a exclude a policy from a container with a specific name? It at least possible with a pod with a specific label, but I would like to exclude some policies for containers with name istio-init and/or istio-proxy

Thanks for any answers :-)

Best regards
Jesper Berggren

[chipzoller]

You should use a precondition using a JMESPath expression that simply skips the policy if a name of your choosing is found.

[chipzoller]

You would have to write the policy as a foreach using a precondition inside the foreach loop if you want to skip the Istio containers but check everything else.

[karlderkaefer]

as said before you need add a condition, they are chained as with logical and, here is my solution (helm syntax)

spec:
  validationFailureAction: {{ .Values.policies.disallow_capabilities_strict.validationFailureAction }}
  background: true
  failurePolicy: {{ .Values.policies.failurePolicy }}
  rules:
    - name: require-drop-all
      match:
        any:
          - resources:
              kinds:
                - Pod
      preconditions:
        all:
          - key: {{ `{{ request.operation || 'BACKGROUND' }}` | quote }}
            operator: NotEquals
            value: DELETE
      validate:
        message: >-
          Containers must drop `ALL` capabilities.
        foreach:
          - list: request.object.spec.[ephemeralContainers, initContainers, containers][]
            deny:
              conditions:
                all:
                  - key: ALL
                    operator: AnyNotIn
                    value: {{ "{{ element.securityContext.capabilities.drop[].to_upper(@) || `[]` }}" | quote }}
    - name: adding-capabilities-strict
      match:
        any:
          - resources:
              kinds:
                - Pod
      preconditions:
        all:
          - key: {{ `{{ request.operation || 'BACKGROUND' }}` | quote }}
            operator: NotEquals
            value: DELETE
      validate:
        message: >-
          Any capabilities added other than NET_BIND_SERVICE are disallowed.
        foreach:
          - list: request.object.spec.[ephemeralContainers, initContainers, containers][]
            deny:
              conditions:
                all:
                  - key: {{ "{{ element.name || '' }}" | quote }}
                    operator: NotEquals
                    value: "istio-init"
                  - key: {{ "{{ element.securityContext.capabilities.add[].to_upper(@) || `[]` }}" | quote }}
                    operator: AnyNotIn
                    value:
                      - NET_BIND_SERVICE
                      - ''

[chipzoller]

@karlderkaefer you may find this Helm escaping syntax slightly easier: https://kyverno.io/docs/writing-policies/variables/#variables-in-helm

[karlderkaefer]

Yes it is not consistent in my example

[AndersBennedsgaard]

@karlderkaefer doesn't this allow any containers and ephemeral containers called istio-init to add capabilities?
I think I've found a solution to this. I tested it using kyverno test with cases for both init-containers and regular containers called istio-init:

rules:
  - name: capabilities
    match:
      resources:
        kinds:
          - Pod
    validate:
      message: >-
        Adding of additional capabilities beyond the default set is not allowed.
        The fields spec.containers[*].securityContext.capabilities.add and
        spec.ephemeralContainers[*].securityContext.capabilities.add must be empty.
      pattern:
        spec:
          containers:
            - =(securityContext):
                =(capabilities):
                  X(add): "null"
          =(ephemeralContainers):
            - =(securityContext):
                =(capabilities):
                  X(add): "null"
          =(initContainers):
            - (name): "!istio-init"
              =(securityContext):
                =(capabilities):
                  X(add): "null"

This does allow the istio-init container to add any capability, and not just a set of selected capabilities, which I haven't found an elegant solution to yet 🤔

[SamStenton]

Has anyone resolved this elegantly? The concern with above is someone could name their container istio-init to get through the block.

[chipzoller]

You should use the image field instead.

[chipzoller]

https://kyverno.io/blog/2024/02/04/securing-services-meshes-easier-with-kyverno/